Advertisement
Original Research| Volume 46, ISSUE 4, P411-418, June 2022

Modified Delphi Process to Identify Research Priorities and Measures for Adult Lifestyle Programs to Address Type 2 Diabetes and Other Cardiometabolic Risk Conditions

Open AccessPublished:January 17, 2022DOI:https://doi.org/10.1016/j.jcjd.2022.01.003

      Abstract

      Objectives

      Clinical and community guidelines recommend lifestyle (i.e. diet and physical activity) interventions for cardiometabolic conditions (including type 2 diabetes), yet current evidence suggests limited and variable services in primary care and public health settings. New implementation research studies are needed to ensure maximal effectiveness, equity and efficiency across all population subgroups and within the context of health systems. Such work will benefit from use of similar core measures and outcome indicators across studies. This Delphi process was undertaken by a new interdisciplinary volunteer researcher network to identify research priorities and core measures for such studies.

      Methods

      Interested network members completed 2 rounds of a modified Delphi process delivered through online questionnaire and teleconferences. Consensus was defined as the median and interquartile range within the top third of a 9-point scale.

      Results

      Twenty-five of 53 (47%) members and 18 (34%) participants completed the round 1 and round 2 surveys, respectively. Of 22 possible research priorities, 4 were rated high priority with consensus, including evaluating the efficacy and effectiveness of interventions in place, improving existing interventions for sustainability and clinical and public health research to advance existing knowledge to develop new capacities. Only 15 of the 93 measures and indicators proposed achieved similar consensus.

      Conclusions

      This first effort confirms broad agreement on research priorities and limited agreement on core indicators/measures. The results provide a starting point for further development of common measures for implementation research in lifestyle studies addressing cardiometabolic conditions.

      Résumé

      Objectifs

      Les lignes directrices cliniques et communautaires recommandent des interventions sur le mode de vie (c.-à-d. le régime alimentaire et l’activité physique) contre les maladies cardiométaboliques (y compris le diabète de type 2), mais les données probantes actuelles font état de services limités et variables dans le cadre des soins primaires et de la santé publique. De nouvelles études en matière de recherche sur la mise en œuvre sont nécessaires pour assurer l’efficacité maximale, l’équité et l’efficience dans tous les sous-groupes de la population et dans le contexte des systèmes de santé. Ces travaux bénéficieront de l’utilisation de mesures et d’indicateurs de résultats de base similaires de toutes les études. Ce processus Delphi a été entrepris par un nouveau réseau interdisciplinaire composé de chercheurs bénévoles pour établir les priorités en matière de recherche et les mesures de base de ces études.

      Méthodes

      Les membres intéressés du réseau ont réalisé 2 cycles du processus Delphi modifié offert sous forme de questionnaire en ligne et par téléconférence. Le consensus a été défini en fonction de la médiane et de l’écart interquartile dans le premier tiers d’une échelle de 9 points.

      Résultats

      Vingt-cinq des 53 (47 %) membres et 18 (34 %) participants ont réalisé, et ce respectivement, les enquêtes de cycle 1 et de cycle 2. Sur les 22 priorités possibles en matière de recherche, 4 se sont vu attribuer une cote de priorité élevée par consensus, soit l’évaluation de l’efficacité et du rendement des interventions en place, l’amélioration des interventions actuelles en matière de viabilité, la recherche clinique et la recherche en santé publique afin de faire progresser les connaissances actuelles pour acquérir de nouvelles compétences. Seules 15 des 93 mesures et indicateurs proposés sont parvenus à un consensus similaire.

      Conclusions

      Cette initiative permet de confirmer le vaste accord sur les priorités en matière de recherche et l’accord limité sur les mesures et indicateurs de base. Les résultats servent de point de départ pour l’élaboration ultérieure de mesures communes de la recherche sur la mise en œuvre d’études sur le mode de vie portant sur les maladies cardiométaboliques.

      Keywords

      Mots clés

      • Researchers agree that new public health and clinical implementation studies are needed to address the high prevalence of cardiometabolic risk conditions.
      • This Delphi study in a newly formed research network for lifestyle approaches identified highest research priorities and possible core measures for such studies.
      • The need for interdisciplinary and intersectoral approaches was highlighted by participants.

      Introduction

      As obesity continues to increase worldwide, cardiometabolic risk (CMR) conditions and diseases, including cardiovascular disease (CVD) and type 2 diabetes, will continue to be major and growing health burdens in many countries (
      Global Burden of Disease 2017 Risk Factor Collaborators
      Global, regional, and national comparative risk assessment of 84 behavioural, environmental and occupational, and metabolic risks or clusters of risks for 195 countries and territories, 1990-2017: A systematic analysis for the Global Burden of Disease Study 2017.
      ). In this context, CMR is being used as a broad term to describe the presence of various combinations of the traditional risk factors combined with features of the metabolic syndrome that contribute to global risk of CVD (
      • Després J.-P.
      • Lemieux I.
      Abdominal obesity and metabolic syndrome.
      ), as well as presence of clinical CVD itself. Excess body weight is associated with heterogeneous metabolic effects, and health risk implications have been variously described by different research groups (
      • Piché M.-E.
      • Poirier P.
      • Lemieux I.
      • Després J.-P.
      Overview of epidemiology and contribution of obesity and body fat distribution to cardiovascular disease: An update.
      ). Adverse metabolic effects of excess body weight become more prominent in middle age, with about 40% of adults >60 years of age having metabolic syndrome (
      Statistics Canada
      Metabolic syndrome in adults, 2012 to 2013.
      ), characterized by 3 or more risk factors, including high waist circumference, high blood pressure, dyslipidemia (defined by low high-density lipoprotein cholesterol and elevated triglyceride levels) and elevated glucose levels (
      • Alberti K.G.
      • Eckel R.H.
      • Grundy S.M.
      • et al.
      Harmonizing the metabolic syndrome: A joint interim statement of the International Diabetes Federation Task Force on Epidemiology and Prevention; National Heart, Lung, and Blood Institute; American Heart Association; World Heart Federation; International Atherosclerosis Society; and International Association for the Study of Obesity.
      ). Although type 2 diabetes is a condition with heterogeneous causes, recent shifts to higher body weight, immigration and aging of the population are contributing to rapid increases in its incidence and prevalence in Canada (see Weisman et al [
      • Weisman A.
      • Fazli G.S.
      • Johns A.
      • Booth G.L.
      Evolving trends in the epidemiology, risk factors, and prevention of type 2 diabetes: A review.
      ] for review). The Canadian Chronic Disease Surveillance System, which uses administrative health-care algorithms to detect diabetes cases, reported 8.8% of Canadians aged ≥1 year were living with diabetes in 2016–2017 (
      • LeBlanc A.G.
      • Jun Gao Y.
      • McRae L.
      • Pelletier C.
      At-a-glance---Twenty years of diabetes surveillance using the Canadian Chronic Disease Surveillance System.
      ). According to the Framingham risk score, one of the current established clinical tools for estimating 10-year CVD risk (
      • Tobe S.W.
      • Stone J.A.
      • Anderson T.
      • et al.
      Canadian Cardiovascular Harmonized National Guidelines Endeavour (C-CHANGE) guideline for the prevention and management of cardiovascular disease in primary care: 2018 update.
      ), 29% of Canadians 20 to 79 years of age were at intermediate or high risk (
      • Hennessy D.A.
      • Tanuseputro P.
      • Tuna M.
      • et al.
      Population health impact of statin treatment in Canada.
      ). Low high-density lipoprotein cholesterol, presence of diabetes and hypertension are major contributing factors to that score.
      Significant progress has been made in understanding the pathophysiologic basis of these conditions, and numerous clinical trials of behaviour change and medications have been conducted. The importance of diet and exercise in moderating these conditions was recognized even in the early CVD trials, such as the Multiple Risk Factor Intervention Trial in the mid-1970s (
      • Dolecek T.A.
      • Milas N.C.
      • Van Horn L.V.
      • et al.
      A long-term nutrition intervention experience: lipid responses and dietary adherence patterns in the Multiple Risk Factor Intervention Trial.
      ), which utilized group and individual counselling and focussed mostly on the treatment of high cholesterol levels, hypertension and smoking (
      • Dolecek T.A.
      • Milas N.C.
      • Van Horn L.V.
      • et al.
      A long-term nutrition intervention experience: lipid responses and dietary adherence patterns in the Multiple Risk Factor Intervention Trial.
      ). Currently, obesity and type 2 diabetes are recognized as additional important conditions to be addressed (
      • Tobe S.W.
      • Stone J.A.
      • Anderson T.
      • et al.
      Canadian Cardiovascular Harmonized National Guidelines Endeavour (C-CHANGE) guideline for the prevention and management of cardiovascular disease in primary care: 2018 update.
      ).
      Currently, all relevant practice guidelines suggest combining lifestyle interventions with medical treatment to prevent and treat CMR conditions (
      • Tobe S.W.
      • Stone J.A.
      • Anderson T.
      • et al.
      Canadian Cardiovascular Harmonized National Guidelines Endeavour (C-CHANGE) guideline for the prevention and management of cardiovascular disease in primary care: 2018 update.
      ,
      • Arnett D.K.
      • Blumenthal R.S.
      • Albert M.A.
      • et al.
      2019 ACC/AHA Guideline on the Primary Prevention of Cardiovascular Disease: Executive Summary: A report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines.
      ,
      National Institute for Health and Care Excellence
      NICE Pathway: Cardiovascular disease prevention overview 2020.
      ,
      Royal Australian College of General Practitioners
      Prevention of vascular and metabolic disease.
      ). However, discrepancies exist between recommended interventions and actual health-care practice, with limited and variable uptake by either primary care or public health (
      • Weisman A.
      • Fazli G.S.
      • Johns A.
      • Booth G.L.
      Evolving trends in the epidemiology, risk factors, and prevention of type 2 diabetes: A review.
      ,
      • Teoh H.
      • Despres J.P.
      • Dufour R.
      • et al.
      Identification and management of patients at elevated cardiometabolic risk in Canadian primary care: How well are we doing?.
      ). Why is uptake poor? Among the many barriers, the high prevalence of CMR conditions, competing priorities, limited evidence for effectiveness in different contexts in Canada (rural and remote; population subgroups), lack of common metrics across studies and lack of capacity and resources have been cited (
      • Pagoto S.
      The current state of lifestyle intervention implementation research: Where do we go next?.
      ).
      The need for interdisciplinary and transdisciplinary dissemination and implementation research to address complex issues in health has been widely recognized (
      • Stokols D.
      • Hall K.L.
      • Taylor B.K.
      • Moser R.P.
      The science of team science: Overview of the field and introduction to the supplement.
      ), with prevention services being one focus (
      • Visscher T.L.
      • Bell C.
      • Gubbels J.S.
      • et al.
      Challenges in lifestyle and community interventions research; a call for innovation.
      ,
      • Ackermann R.T.
      • O'Brien M.J.
      Evidence and challenges for translation and population impact of the diabetes prevention program.
      ). Many of the resources for lifestyle change in communities, such as self-monitoring devices, a more diverse food supply and recreation centres and parks, already exist but have not necessarily been focussed on addressing CMR conditions. Determining the most effective and efficient strategies to address CMR conditions and diseases in Canada is not straightforward. As with other complex health issues, like mental health and addictions, various combinations of community, workplace, public health and treatment approaches may be most effective and efficient. A wide range of academic and professional expertise is needed in areas such as nutrition, kinesiology, psychology, behaviour change, social work, family medicine, public health medicine, nursing, epidemiology, political science, policy analysis, digital communications, project management, implementation science and health economics to find the best mix of services. To encourage more such collaborative and translational research, individual researchers from different disciplinary backgrounds need to come together, develop proposals, obtain funding and test various approaches. In the past, such teams have often been situated within academic institutions or discipline professional societies or sponsored by disease-focussed nongovernmental organizations (
      • Hall J.G.
      A meeting of minds: Interdisciplinary research in the health sciences in Canada.
      ). New interactive online communication tools offer potential opportunities to develop new types of networks and teams to tackle these more complex issues.
      How to create and sustain such larger teams is far from clear, however. In Canada, there has been some funding support to develop team proposals for dissemination activities and for interdisciplinary training of students and junior researchers and professionals (
      • Gillis D.
      • Nelson J.
      • Driscoll B.
      • Hodgins K.
      • Fraser E.
      • Jacobs S.
      Interdisciplinary and transdisciplinary research and education in Canada: A review and suggested framework.
      ). Learning from efforts to date, and the emergence of “team science,” the study of how to promote effective science teams offers practical insight and context for this Delphi study (
      • Begerowski S.R.
      • Traylor A.M.
      • Shuffler M.L.
      • Salas E.
      An integrative review and practical guide to team development interventions for translational science teams: One size does not fit all.
      ). Aarons et al (
      • Aarons G.A.
      • Reeder K.
      • Miller C.J.
      • Stadnick N.A.
      Identifying strategies to promote team science in dissemination and implementation research.
      ) reported that a small group of dissemination and implementation researchers agreed that 3 strategies were critical at the developmental stage of interdisciplinary and transdisciplinary teams: clear expectations/roles, effective communications and shared goals/mission. Within these broad strategies, several specific strategies were mentioned that are relevant to this project, including shared mental models, consensus building, building trust and common language.
      To begin to develop shared goals requires work to identify research priorities, using measures and outcomes that will be valid and comparable over time. Previous research and literature reviews of lifestyle for CMR have indicated little consistency in research priorities or core outcomes. To begin to achieve this objective of shared goals and within the constraints of online interaction, Delphi consensus processes offer an approach that is transparent and can be documented. Within the broad topic of CMR, 2 recent examples of Delphi processes were found. Wu et al (
      • Wu N.
      • O'Reilly S.
      • Nielsen K.K.
      • Maindal H.T.
      • Dasgupta K.
      Core outcome set for diabetes after pregnancy prevention across the life span: International Delphi study.
      ) aimed to develop a set of core outcomes for studies of diabetes prevention after pregnancy among women who had gestational diabetes, and Stoner et al (
      • Stoner L.
      • Matheson A.G.
      • Perry L.G.
      • et al.
      Principles and strategies for improving the prevention of cardio-metabolic diseases in indigenous populations: An international Delphi study.
      ) had experts from 3 countries rank historic, economic and sociocultural factors contributing to CMR among Indigenous peoples. In our context, a Delphi process was undertaken to identify initial areas and degree of consensus on potential research priorities and measures for lifestyle implementation studies in CMR conditions among a newly formed volunteer researcher network.

      Methods

      We employed modified Delphi techniques to achieve our objective. Originally developed by the RAND Corporation (
      • Fitch K.
      • Bernstein S.J.
      • Aguilar M.D.
      • Burnand B.
      • LaCalle J.R.
      The RAND/UCLA Appropriateness Method User's Manual.
      ), this method has evolved over time, with numerous variations (
      • Junger S.
      • Payne S.A.
      • Brine J.
      • Radbruch L.
      • Brearley S.G.
      Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review.
      ). Online survey methods and digital technology for discussions offer new opportunities to engage diverse participants. The literature on previous research prioritization (
      Working Group on Priority Setting
      Priority setting for health research: Lessons from developing countries.
      ,
      • Viergever R.F.
      • Olifson S.
      • Ghaffar A.
      • Terry R.F.
      A checklist for health research priority setting: Nine common themes of good practice.
      ) and indicator efforts was reviewed, as multiple groups have been undertaking such efforts, using a range of approaches from consensus by an invited group of experts to formal modified Delphi processes, to identify both areas of agreement and divergent views. A 2-round modified virtual Delphi process was planned and conducted from February to April 2020.

      Participant recruitment

      As a first step, funding was obtained to initiate a new network of researchers to promote the development of new research proposals and identify parameters for potential implementation proposals (see Figure 1). Potentially interested researchers and implementation experts from Canada, the United States, the United Kingdom and Australia were identified from personal contacts and from discussion at 2 events in 2019: the Diabetes Canada and North American Primary Care Research Group conferences. In addition, all researchers active in Canada who had published on primary care and/or lifestyle interventions in diabetes, prediabetes and metabolic syndrome over the past 10 years were identified through PubMed and CINAHL and contacted by email for possible interest in the network. To provide attendees with the background and goals of the network, describe the process and create an opportunity to engage, 2 teleconferences and a webinar on knowledge translation were completed before initiating the modified Delphi process. Invitation emails were sent to 53 network members with a link to the survey and an attached document describing the purpose and process, as well as shared access (through Dropbox) to 6 key background articles (
      • Viergever R.F.
      • Olifson S.
      • Ghaffar A.
      • Terry R.F.
      A checklist for health research priority setting: Nine common themes of good practice.
      ,
      • Alligier M.
      • Barres R.
      • Blaak E.E.
      • et al.
      OBEDIS Core Variables Project: European expert guidelines on a minimal core set of variables to include in randomized, controlled clinical trials of obesity interventions.
      ,
      • Hales S.
      • Lesher-Trevino A.
      • Ford N.
      • Maher D.
      • Ramsay A.
      • Tran N.
      Reporting guidelines for implementation and operational research.
      ,
      • Maclean P.S.
      • Rothman A.J.
      • Nicastro H.L.
      • et al.
      The Accumulating Data to Optimally Predict Obesity Treatment (ADOPT) core measures project: Rationale and approach.
      ,
      • Pinnock H.
      • Barwick M.
      • Carpenter C.R.
      • et al.
      Standards for Reporting Implementation Studies (StaRI) statement.
      ,
      • Rudan I.
      Setting health research priorities using the CHNRI method: IV. Key conceptual advances.
      ). Members were also encouraged to send an invitation to others interested.

      Questionnaire development

      Evaluation criteria and research priorities

      A preliminary list of research priorities was generated through feedback from network members from the first 2 teleconferences (n=53). The list was highly diverse and included a mixture of specific and general topics, reflecting the many ways research can be described. The literature was then searched for possibly relevant knowledge translation frameworks (
      • Strifler L.
      • Cardoso R.
      • McGowan J.
      • et al.
      Scoping review identifies significant number of knowledge translation theories, models, and frameworks with limited use.
      ), but research prioritization was rarely mentioned. Next, the global health literature was explored. The work of the Child Health and Nutrition Research Initiative (CHNRI) was notable as they had used systematic and transparent methods to set child-focussed health systems research priorities in low- and middle-income countries (
      • Black R.E.
      The legacy of the Child Health and Nutrition Research Initiative (CHNRI).
      ). Multiple health research prioritization exercises had been undertaken over several years using CHNRI methods and 2 of their innovations were adapted to the present project. The first was an explicit list of priority-setting criteria that may apply to judgements of different stakeholders in a research priority-setting context. These 20 evaluation criteria were rated for relevance to our topic area (
      • Rudan I.
      Setting health research priorities using the CHNRI method: IV. Key conceptual advances.
      ). The second innovation was the use of the 4D framework for describing research focus (“description,” “delivery,” “development” and “discovery” research), which we adapted to describe 22 specific research areas for assessment (
      • Rudan I.
      Setting health research priorities using the CHNRI method: IV. Key conceptual advances.
      ).

      Indicators and measures

      Although international efforts are underway to establish core outcome sets for studies in different diseases, CMR conditions have not yet been reviewed by either the International Consortium for Health Outcomes Measurements (
      International Consortium for Health Outcome Measurement. 2022
      ) or the Core Outcome Measures in Effectiveness Trials database (
      COMET Initiative
      Core Outcome Measures in Effectiveness Trials.
      ). As our focus was lifestyle in CMR conditions, we chose to focus on previous consensus work in obesity as a starting point, on the assumption that key indicators and outcomes for lifestyle interventions would be identified, because lifestyle interventions are well established in obesity. Measures from 2 different consensus processes addressing obesity management were found, categorized, combined and adapted by 1 researcher (P.B.) to create an initial list. Some measures were identified by both groups, but many were not. The Accumulating Data to Optimally Predict Obesity Treatment (ADOPT) is a United States–based effort to better understand and deliver tailored obesity treatments, based on behavioural, biologic, environmental and psychosocial domains. Part of this work involved the development of a core list of constructs and measures, an initial list of high-priority constructs based on the current state of science (
      • Maclean P.S.
      • Rothman A.J.
      • Nicastro H.L.
      • et al.
      The Accumulating Data to Optimally Predict Obesity Treatment (ADOPT) core measures project: Rationale and approach.
      ). The OBEsity Diverse Interventions Sharing (OBEDIS) group developed a set of European guidelines for a minimal set of variables to include in future clinical trials on obesity, regardless of the specific endpoints (
      • Alligier M.
      • Barres R.
      • Blaak E.E.
      • et al.
      OBEDIS Core Variables Project: European expert guidelines on a minimal core set of variables to include in randomized, controlled clinical trials of obesity interventions.
      ). In addition, other possible measures were identified from previous implementation study guides (
      • Hales S.
      • Lesher-Trevino A.
      • Ford N.
      • Maher D.
      • Ramsay A.
      • Tran N.
      Reporting guidelines for implementation and operational research.
      ,
      • Pinnock H.
      • Barwick M.
      • Carpenter C.R.
      • et al.
      Standards for Reporting Implementation Studies (StaRI) statement.
      ). The final 7 categories were as follows:
      • Demographic/environment/context (15 indicators/measures).
      • Diet and physical activity (8 indicators/measures).
      • Formative and process measures (6 indicators/measures).
      • Psychosocial constructs (21 indicators/measures).
      • Body composition and energy estimation (6 indicators/measures).
      • Biological/Clinical/History Measures (33 indicators/measures).
      • Social Environment and Context (4 indicators/measures).
      The finalized online modified Delphi survey was divided into 3 sections: Evaluation Criteria (for research priorities), Research Priorities and Measures/Constructs/Outcomes. Each statement was rated on a 9-point scale, ranging from “Not relevant/not priority/not core” (score 1 to 3), to “Uncertain/equivocal” (score 4 to 6), to “Highly relevant/priority/core” (score 7 to 9). The online survey avoided making any forced responses and provided forward/backward buttons. Additional suggestions and comments were requested at the end of each section to capture possible additional indicators. The survey was reviewed and pilot-tested by the core research team (P.B., S.D., R.D.). The round 1 and round 2 surveys were very similar, with 132 and 135 statements, respectively. Based on feedback from round 1, in round 2, there were 2 indicators added to the “Demographic/environment/context” category (i.e. marital status and material hardship; e.g. difficulty paying bills, rent, etc) and 1 indicator was added to the “Diet and physical activity” category (i.e. type/extent of food processing as an indicator of diet quality). The final modified Delphi questionnaire is available in the Supplementary Material.

      Data collection

      Each round of the modified Delphi process was delivered as a self-administered survey (developed and delivered through an online survey software platform (Qualtrics, Seattle, Washington, United States) and debriefing virtual meetings. Six relevant background studies selected by the core research team were made available to all participants through the Dropbox sharing feature (
      • Viergever R.F.
      • Olifson S.
      • Ghaffar A.
      • Terry R.F.
      A checklist for health research priority setting: Nine common themes of good practice.
      ,
      • Alligier M.
      • Barres R.
      • Blaak E.E.
      • et al.
      OBEDIS Core Variables Project: European expert guidelines on a minimal core set of variables to include in randomized, controlled clinical trials of obesity interventions.
      ,
      • Hales S.
      • Lesher-Trevino A.
      • Ford N.
      • Maher D.
      • Ramsay A.
      • Tran N.
      Reporting guidelines for implementation and operational research.
      ,
      • Maclean P.S.
      • Rothman A.J.
      • Nicastro H.L.
      • et al.
      The Accumulating Data to Optimally Predict Obesity Treatment (ADOPT) core measures project: Rationale and approach.
      ,
      • Pinnock H.
      • Barwick M.
      • Carpenter C.R.
      • et al.
      Standards for Reporting Implementation Studies (StaRI) statement.
      ,
      • Rudan I.
      Setting health research priorities using the CHNRI method: IV. Key conceptual advances.
      ). For round 1, participants had about 2 weeks to complete the survey (February 25 to March 12, 2020). Before round 2, they received an Excel summary listing each statement, the group results (median and 25th to 75th percentile) and their own answers. Two virtual meetings were conducted to allow for differences in participant schedules. Both meetings had the same format of reviewing the overall results and most contentious statements, as well as a general discussion of the process. Meeting notes were reviewed by the research team and sent to participants after each meeting. For round 2, the survey was deployed with a 2-week turnaround (March 28 to April 12, 2020) and results were again sent back to participants. A final online conference meeting was held in April 2020 to review results and discuss next steps for the development of the network.

      Analysis

      Initial review of group results after round 1 indicated that all but 1 statement (Evaluation Criteria: “Likelihood of generating patents/lucrative products”) was at least partially supported, so scores were highly skewed toward a higher score. Despite encouragement to use the full scoring range, scores were still highly skewed after round 2. To report results, we therefore used the most strict definition of strong agreement, as described by the original developers (
      • Fitch K.
      • Bernstein S.J.
      • Aguilar M.D.
      • Burnand B.
      • LaCalle J.R.
      The RAND/UCLA Appropriateness Method User's Manual.
      ), which required that all 3 indicators (median, 25th percentile and 75th percentile) fell within the 1 to 3 (strong negative) or 7 to 9 (strong positive) scoring range. A second set of statements had the median and 75th percentile fall within 7 to 9 range, but the 25th percentile indicated more variable scoring. These statements achieved weaker agreement. The third group of uncertain statements had lower medians and diverse ranges.

      Results

      Participation

      A total of 53 researchers from Canada (n=47), the United States (n=3), the United Kingdom (n=1) and Australia (n=2) expressed interest in the network, and 50 of them were interested in participating in the modified Delphi process. Only 1 member of the research team (P.B.), who created the initial list, did not complete the modified Delphi questionnaire. Of the 53 people contacted by email about the modified Delphi process, 47% (n=25, including 1 from the United States, 2 from British Columbia, 3 from Alberta, 1 from Saskatchewan, 3 from Manitoba, 5 from Ontario, 8 from Québec and 2 from Newfoundland) completed the round 1 modified Delphi survey and attended round 1 online meetings; 34% (n=18, including 1 from United States, 1 from British Columbia, 2 from Alberta, 1 from Saskatchewan, 1 from Manitoba, 5 from Ontario, 5 from Québec and 2 from Newfoundland) completed the round 2 modified Delphi survey and attended the round 2 online meeting. Average time to completion of the 2 surveys was 32.6 and 30.3 minutes, respectively.

      Modified Delphi process

      After round 2, strong agreement was achieved on 7 of the 20 Evaluation Criteria, including “Answerability,” “Potential for translation,” “Effectiveness,” “Maximum potential impact on the burden,” “Equity,” “Relevance” and “Fills key gap” (Table 1). Seven additional criteria had more variance, and only the “Likelihood of generating patents/lucrative products” had a low median (median, 25th to 75th percentile, 1 to 5).
      Table 1Agreement with evaluation criteria
      Strong agreement statement
      All 3 indicators (median, 25th percentile and 75th percentile) fall between 7 and 9.
      Weaker agreement statement
      The 25th percentile falls outside the 7 to 9 range.
      AnswerabilityNovelty
      Potential for translationDeliverability
      EffectivenessSustainability
      Maximum potential impact on the burdenEthical aspects
      EquityCommunity involvement
      RelevanceFeasibility
      Fills key gapFundability
      Note: Criteria rated as uncertain (did not meet the criteria for strong or weaker agreement) were “Attractiveness” (some health research ideas will be more likely to lead to publications in high-impact journals), “Affordability” (the implementation of knowledge generated through some health research ideas will not be affordable within the context), “Public opinion,” “Cost,” “Alignment with political priorities” and “Likelihood of generating patents/lucrative products.”
      All 3 indicators (median, 25th percentile and 75th percentile) fall between 7 and 9.
      The 25th percentile falls outside the 7 to 9 range.
      Among 22 Research Priorities, 4 consensus priorities emerged, as shown in Table 2. An additional 9 priorities had greater variance, but none were low priority (median, 1 to 3).
      Table 2Agreement with research priorities
      Strong agreement statement
      All 3 indicators (median, 25th percentile and 75th percentile) fall between 7 and 9.
      Weaker agreement statement
      The 25th percentile falls outside the 7 to 9 range.
      Evaluating the efficacy and effectiveness of interventions in placeUnderstanding risk factors

      Measuring prevalence of exposure to risk factors
      Improving existing interventions for sustainabilityMeasuring prevalence of coverage of interventions in place
      Clinical research to advance existing knowledge to develop new capacities

      Public health research to advance existing knowledge to develop new capacities
      Improving existing interventions for affordability

      Improving existing interventions for deliverability

      Improving existing interventions for acceptability

      Basic research to advance existing knowledge to develop new capacities
      Clinical research to explore entirely novel ideas
      Public health research to explore entirely novel ideas
      Note: Priorities rated as uncertain/equivocal (did not meet the criteria for strong or weaker agreement) were “Measuring the burden,” “Health policy analysis,” “Health system structure analysis,” “Financing/costs analysis,” “Human resources,” “Provision/infrastructure,” “Operations research,” “Responsiveness of recipients” and “Basic research to explore entirely novel ideas.”
      All 3 indicators (median, 25th percentile and 75th percentile) fall between 7 and 9.
      The 25th percentile falls outside the 7 to 9 range.
      Fifteen of the 93 “Measures/constructs/outcomes” achieved high-priority consensus: “Demographic/environment/context” (n=4), “Diet and physical activity” (n=2), “Formative and process measures” (n=1), “Psychosocial constructs” (n=1), “Body composition” (n=3) and “Biologic/clinical history measures” (n=4). None were considered of low relevance (median, 1 to 3), as shown in Table 3. Nine statements were categorized as having weaker agreement; for example, “Gender (categorical)” and various statements about physical activity fell into this group.
      Table 3Agreement with measures/constructs/outcomes
      Strong agreement
      All 3 indicators (median, 25th percentile and 75th percentile) fall between 7 and 9.
      Weaker agreement
      The 25th percentile falls outside the 7 to 9 range.
      Uncertain
      All others.
      Demographic/environment/context
       Ethnicity

      Indigenous identity

      Education

      Income
      Gender score (categorical)

      Access to health care
      Gender score (continuous)

      Alcohol use

      Tobacco use

      Language

      Work status

      Marital status

      Material hardship

      Household size

      Health insurance status
      Diet and physical activity
       Usual dietary intake from 24-hour recalls or food frequency questionnaire

      Diet quality
      Physical activity record/diary

      Self-report physical activity questionnaire

      Physical activity, sedentary activity, sleep from actigraphy or accelerometer

      Sedentary behaviours by a questionnaire
      Type/extent of food processing as indicator of diet quality

      Steps walked/day record
      Formative and process measures
       Intervention experience of participants in programNAIncremental cost-effectiveness ratio

      Costing of intervention

      Intervention fidelity by activity reporting of interventionists

      Intervention fidelity by observation/recording

      Documentation of involvement of the public in developing the intervention
      Psychosocial constructs
       Quality-of-life questionnaire addressing self-care, usual activities, pain/discomfort and anxiety/depressionGeneral health or well-being

      Self-efficacy
      Medication adherence

      Perceived stress

      Trait Anxiety Inventory

      Depression

      Intuitive Eating Scale

      Three-factor Eating Questionnaire

      Self-weighing behaviour

      State affect

      Emotional eating

      Binge eating

      Trait food craving

      Reward-related eating

      Executive function

      Delay discounting

      Behavioural intention

      Hedonic responses

      Hunger and satiety

      Personality: Big 5 factors
      Body composition and energy

      estimation
       Anatomic fat storageNAExpended energy
       Weight/height/BMI/waist circumference

      Energy intake (overlap with diet measures)
      Energy expenditure---activity

      Energy expenditure---resting metabolic rate
      Biologic/clinical history measures
       A1C

      Lipid profile

      Comorbidity score

      Blood pressure
      Menopausal statusCardiorespiratory fitness at rest---resting heart rate

      Cardiorespiratory fitness---walk test

      VO2max---indirect
      Muscle strength from handgrip strength
      Glucose area under curve 2 hours postprandial
      Insulin, glucagon, HOMA-IR, HOMA-B
      Nutrient status
      Fasting glucose (2 times)
      ApoA; apoB
      Cardiovascular disease risk score
      Liver function (aminotransferases [U/L], alkaline phosphatase [U/L], gamma-glutamyl transpeptidase [U/L])
      Liver function---NFS and FIB4---fibrosis; platelet count; albumin
      Fatty liver---ultrasound
      Inflammation (IL-6, TNF-alpha, CRP)
      High-sensitivity C-reactive protein
      Heart electrical activity (ECG)
      Self-report of sleep duration and timing
      Sleep disorders (apnea)
      Age of onset of obesity
      Maximal and minimum weight after age 18 years
      Variation in body weight in past 3 months
      Previous attempts at weight loss and weight maintained after loss
      Etiology---genetic heritage
      Parental weight history, including maternal bariatric surgery
      Family history of diabetes
      Hunger/satiety---ghrelin, GLP-1, PYY3-36 (panel)
      Energy homeostasis---adipose tissue-derived hormones (adiponectin, leptin, amylin)
      Thyroid hormones (TSH, T4, fT4I (panel)
      Social environment and contextBMI of others in social network, objective
      Weight status of individuals in social network, perceived
      Support from social network
      Perceived autonomy support
      A1C, glycated hemoglobin; Apo, apolipoprotein; BMI, body mass index; CRP, C-reactive protein; ECG, electrocardiogram; FIB4, specific liver fibrosis index; fT4I, free thyroxine (T4) index; GLP-1, glucagon-like peptide-1; HOMA, homeostatic model assessment (of insulin resistance or β cell function); IL-6, interleukin-6; NFS, non-alcoholic fatty liver disease fibrosis score; PYY3-36, peptide YY; T4, thyroxine; TNF, tumour necrosis factor; TSH, thyroid-stimulating hormone.
      All 3 indicators (median, 25th percentile and 75th percentile) fall between 7 and 9.
      The 25th percentile falls outside the 7 to 9 range.
      All others.

      Participants’ comments and discussion

      Participants’ diverse perspectives were evident throughout the process. Varied opinions were identified on diet and physical assessments, and approaches to assess CMR conditions, in line with issues of feasibility, measurement error, validity and cost based on participants’ own research foci and experiences. Several participants emphasized the importance of involving relevant stakeholders (e.g. community partners and policymakers) at all stages of planned research proposal development.
      Many participants were not familiar with the modified Delphi process, or with some of the measures and indicators. There were some missing data, as some experts were uncomfortable rating measures for which they were unfamiliar. Terminology for the anchors (i.e. “Relevant” or “Not relevant”) for the 9-point scales was problematic for a few participants. In discussion after the first survey, anchors for the sections on Evaluation Criteria and Measures/Constructs/Outcomes were changed to “Core” or “Not core.”

      Discussion

      The modified Delphi consensus process is an increasingly popular method to document the degree of agreement for complex topics. To our knowledge, this was the first attempt to use this method to establish research priorities and core measures and indicators specifically directed to lifestyle implementation studies for CMR conditions. Participants reached consensus on 4 of 22 research priorities (Table 2), indicating most of them agreed that efforts in our own countries should be directed to uptake/scale-up of current interventions in both the clinical and the public health systems.
      The addition of explicit consideration of the 20 evaluation criteria was prompted by previous work by the CHNRI in obtaining feedback on priorities from diverse experts (
      • Rudan I.
      Setting health research priorities using the CHNRI method: IV. Key conceptual advances.
      ). Despite our high-income setting, we felt we may have an analogous situation in which researchers from different disciplinary backgrounds were coming together, and the top-ranked criteria were aligned quite closely with the 5 CHNRI high-priority “standard” criteria, namely answerability, effectiveness, deliverability, maximum potential for disease burden reduction and the effect on equity. Our results differ from the CHNRI criteria in terms of interventions being deliverable, which likely reflects the greater focus on resource and infrastructure challenges in low- and middle-income countries. Our participants agreed that research with strong potential for translation was core. Our group also agreed that research that could fill a key gap was core, defined as “health research ideas that are more likely to fill the key gap in knowledge that is required for translation and/or implementation than the others.” Generally, participants in this work had quite similar evaluation criteria.
      There was less consensus with respect to outcomes, indicators and measures. Our interest in measures of CMR conditions differed from 2 other consensus processes for measuring obesity interventions (ADOPT and OBEDIS) (
      • Alligier M.
      • Barres R.
      • Blaak E.E.
      • et al.
      OBEDIS Core Variables Project: European expert guidelines on a minimal core set of variables to include in randomized, controlled clinical trials of obesity interventions.
      ,
      • Rosenbaum M.
      • Agurs-Collins T.
      • Bray M.S.
      • et al.
      Accumulating Data to Optimally Predict Obesity Treatment (ADOPT): Recommendations from the biological domain.
      ). The ADOPT measures were focussed on explanatory research in weight loss (
      • Maclean P.S.
      • Rothman A.J.
      • Nicastro H.L.
      • et al.
      The Accumulating Data to Optimally Predict Obesity Treatment (ADOPT) core measures project: Rationale and approach.
      ), whereas OBEDIS goals were closer to our own in recommending fewer core measures, but they were still oriented to weight-loss treatment (
      • Alligier M.
      • Barres R.
      • Blaak E.E.
      • et al.
      OBEDIS Core Variables Project: European expert guidelines on a minimal core set of variables to include in randomized, controlled clinical trials of obesity interventions.
      ). Implementation variables, such as formative and process evaluation measures, were not included in either of the earlier efforts. The reasons for variability in the scoring of some measures, such as gender and physical activity, warrant further investigation. Additional work on core measures is needed with a larger study group, with additional opportunities to discuss strengths and limitations of various measures. To what extent did the process contribute to our overall goal of developing a common set of goals/mission? Interestingly, some clear research priorities did emerge, and interest in further developing the research network was expressed, yet it also became clear that much more work is needed and with a larger group to develop an agreed-upon set of outcomes and measures.

      Strengths and limitations

      Strengths of this study consisted of an inclusive and transparent process completed in a timely fashion using technology that allowed for both qualitative and quantitative feedback among volunteers in an emerging research network. The use of strict criteria for consensus made it possible to discern a small core set of research priorities for CMR conditions that have not been documented previously.
      Although considered superior to typical committee approaches, in which one or more person/people may dominate decision-making, the modified Delphi approach represents the opinions of the participants involved (
      • Keeney S.
      • Hasson F.
      • McKenna H.
      Consulting the oracle: Ten lessons from using the Delphi technique in nursing research.
      )---in the present case limited to mainly Canadian researchers attracted to the topic, who had been either invited by email based on first authorship in a peer-reviewed article cited in PubMed, and/or through conversations with already involved researchers. Certainly, network participants are a convenience sample of the relevant research communities. Although we did not collect information on participants’ disciplinary backgrounds and research focus, a sense of the diverse interests of participants can be gained by reviewing the author list and the information posted to the Cardiometabolic Research Network webpage of the Canadian Nutrition Society (https://cns-scn.ca/education-resources/cardiometabolic-research-network-cmrn/who-we-are). Membership is free and members have access to the contact information of all other members in the network. Additional approaches, such as our workshop on methods of engaging patients, will be needed to bring together the wider spectrum of stakeholders, including patients, health promotion and health-care professionals, health managers, funders and researchers, to confirm and extend this work. Another limitation is the lack of availability of a systematic review on all possible measures and indicators. Use of the results of 2 previous consensus processes in obesity was conceptually justified by the overlap in measures in lifestyle studies of obesity and CMR conditions. Both previous processes, however, were focussed on treatment and not prevention or community interventions outside of health care. A systematic literature review is needed, ideally including patients and other stakeholders, to ensure that the entire spectrum of relevant measures and indicators can be considered. Completion and publication of this first effort provides a starting point for such a review.
      Other challenges and limitations included the fact that only half of the group started the modified Delphi process and fewer still were able to complete both surveys. COVID-19 shutdowns and changes in March 2020 affected the academic and personal lives of many in this group, which may have had an impact on participation (
      Public Health Agency of Canada
      Coronvirus disease (COVID-19).
      ). Future consensus efforts should pay attention to allowing enough time for orientation to the modified Delphi process and more extensive discussion opportunities after the first round of ratings so the strengths and limitations of various measures can be reviewed. The time commitment for this process, with planned meetings, was likely a barrier to participation.
      To our knowledge, this study is the first to investigate research priorities, core measures and indicators for lifestyle implementation studies for CMR conditions. Our findings may be helpful to researchers interested in developing lifestyle-oriented research proposals. Through 2 rounds of the modified Delphi process, we identified that current research priorities should focus on the implementation or scale-up of existing interventions in both primary health-care and community settings. Although less consensus was achieved in evaluating measures/constructs/outcomes, the relevant concepts will need to be re-examined with a larger participation group to distinguish the most essential measures from the rest of the list.
      We also identified that the virtual modified Delphi process can serve as an efficient method for generating research priories for newly developing research networks in determining the extent to which members have common goals and priorities. More work is needed to identify additional successful strategies for further developing common goals and measures in research teams when the group begins with limited consensus, as in the present study. Multiple ideas are being suggested in the team sciences literature (
      • Begerowski S.R.
      • Traylor A.M.
      • Shuffler M.L.
      • Salas E.
      An integrative review and practical guide to team development interventions for translational science teams: One size does not fit all.
      ,
      • Aarons G.A.
      • Reeder K.
      • Miller C.J.
      • Stadnick N.A.
      Identifying strategies to promote team science in dissemination and implementation research.
      ); the next steps will involve obtaining funding to promote development of common goals for the network.

      Acknowledgments

      The kind support from Metabolic Syndrome Canada and the Canadian Nutrition Society is gratefully acknowledged. Funding was obtained from a CIHR Planning and Dissemination Grant. The funder had no role in the design of the study, data collection and analysis or interpretation of the data or writing of the manuscript.

      Author Disclosures

      Conflicts of interest: None.

      Author Contributions

      P.B., S.D. and R.D. initiated the network, obtained funding and guided the process throughout. P.B. and Y.W. created the online questionnaire, analyzed results and wrote initial drafts. A.L. helped initiate the network. A.I.C., D.K., J.-P.D.-C., J.R., J.R.K., M.L.P., R.B., S.V.H., Z.H. and S.S. are network members who participated in the process and reviewed and revised the article. All authors read and approved the final version of the manuscript submitted for publication.

      Supplementary Material

      References

        • Global Burden of Disease 2017 Risk Factor Collaborators
        Global, regional, and national comparative risk assessment of 84 behavioural, environmental and occupational, and metabolic risks or clusters of risks for 195 countries and territories, 1990-2017: A systematic analysis for the Global Burden of Disease Study 2017.
        Lancet. 2018; 392: 1923-1994
        • Després J.-P.
        • Lemieux I.
        Abdominal obesity and metabolic syndrome.
        Nature. 2006; 444: 881-887
        • Piché M.-E.
        • Poirier P.
        • Lemieux I.
        • Després J.-P.
        Overview of epidemiology and contribution of obesity and body fat distribution to cardiovascular disease: An update.
        Progr Cardiovasc Dis. 2018; 61: 103-113
        • Statistics Canada
        Metabolic syndrome in adults, 2012 to 2013.
        Statistics Canada. 2015., Ottawa2015
        • Alberti K.G.
        • Eckel R.H.
        • Grundy S.M.
        • et al.
        Harmonizing the metabolic syndrome: A joint interim statement of the International Diabetes Federation Task Force on Epidemiology and Prevention; National Heart, Lung, and Blood Institute; American Heart Association; World Heart Federation; International Atherosclerosis Society; and International Association for the Study of Obesity.
        Circulation. 2009; 120: 1640-1645
        • Weisman A.
        • Fazli G.S.
        • Johns A.
        • Booth G.L.
        Evolving trends in the epidemiology, risk factors, and prevention of type 2 diabetes: A review.
        Can J Cardiol. 2018; 34: 552-564
        • LeBlanc A.G.
        • Jun Gao Y.
        • McRae L.
        • Pelletier C.
        At-a-glance---Twenty years of diabetes surveillance using the Canadian Chronic Disease Surveillance System.
        Health Promot Chron Dis Prev Can. 2019; 39: 306-309
        • Tobe S.W.
        • Stone J.A.
        • Anderson T.
        • et al.
        Canadian Cardiovascular Harmonized National Guidelines Endeavour (C-CHANGE) guideline for the prevention and management of cardiovascular disease in primary care: 2018 update.
        CMAJ. 2018; 190: E1192-e1206
        • Hennessy D.A.
        • Tanuseputro P.
        • Tuna M.
        • et al.
        Population health impact of statin treatment in Canada.
        Health Rep. 2016; 27: 20-28
        • Dolecek T.A.
        • Milas N.C.
        • Van Horn L.V.
        • et al.
        A long-term nutrition intervention experience: lipid responses and dietary adherence patterns in the Multiple Risk Factor Intervention Trial.
        J Am Diet Assoc. 1986; 86: 752-758
        • Arnett D.K.
        • Blumenthal R.S.
        • Albert M.A.
        • et al.
        2019 ACC/AHA Guideline on the Primary Prevention of Cardiovascular Disease: Executive Summary: A report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines.
        Circulation. 2019; 140: e563-e595
        • National Institute for Health and Care Excellence
        NICE Pathway: Cardiovascular disease prevention overview 2020.
        • Royal Australian College of General Practitioners
        Prevention of vascular and metabolic disease.
        in: Guidelines for Preventive Actitivies in General Practice. 9th ed. Royal Australian College of General Practitioners, East Melbourne, Victoria2018
        • Teoh H.
        • Despres J.P.
        • Dufour R.
        • et al.
        Identification and management of patients at elevated cardiometabolic risk in Canadian primary care: How well are we doing?.
        Can J Cardiol. 2013; 29: 960-968
        • Pagoto S.
        The current state of lifestyle intervention implementation research: Where do we go next?.
        Transl Behav Med. 2011; 1: 401-405
        • Stokols D.
        • Hall K.L.
        • Taylor B.K.
        • Moser R.P.
        The science of team science: Overview of the field and introduction to the supplement.
        Am J Prev Med. 2008; 35: S77-S89
        • Visscher T.L.
        • Bell C.
        • Gubbels J.S.
        • et al.
        Challenges in lifestyle and community interventions research; a call for innovation.
        BMC Obes. 2014; 1: 29
        • Ackermann R.T.
        • O'Brien M.J.
        Evidence and challenges for translation and population impact of the diabetes prevention program.
        Curr Diab Rep. 2020; 20: 9
        • Hall J.G.
        A meeting of minds: Interdisciplinary research in the health sciences in Canada.
        CMAJ. 2006; 175: 763-771
        • Gillis D.
        • Nelson J.
        • Driscoll B.
        • Hodgins K.
        • Fraser E.
        • Jacobs S.
        Interdisciplinary and transdisciplinary research and education in Canada: A review and suggested framework.
        Collect Essays Learn Teach. 2017; 10: 203-222
        • Begerowski S.R.
        • Traylor A.M.
        • Shuffler M.L.
        • Salas E.
        An integrative review and practical guide to team development interventions for translational science teams: One size does not fit all.
        J Clin Transl Sci. 2021; 5: 1-31
        • Aarons G.A.
        • Reeder K.
        • Miller C.J.
        • Stadnick N.A.
        Identifying strategies to promote team science in dissemination and implementation research.
        J Clin Transl Sci. 2020; 4: 180-187
        • Wu N.
        • O'Reilly S.
        • Nielsen K.K.
        • Maindal H.T.
        • Dasgupta K.
        Core outcome set for diabetes after pregnancy prevention across the life span: International Delphi study.
        BMJ Open Diab Res Care. 2020; 8e001594
        • Stoner L.
        • Matheson A.G.
        • Perry L.G.
        • et al.
        Principles and strategies for improving the prevention of cardio-metabolic diseases in indigenous populations: An international Delphi study.
        Prev Med. 2017; 96: 106-112
        • Fitch K.
        • Bernstein S.J.
        • Aguilar M.D.
        • Burnand B.
        • LaCalle J.R.
        The RAND/UCLA Appropriateness Method User's Manual.
        RAND Corporation, Santa Monica, CA2001
        • Junger S.
        • Payne S.A.
        • Brine J.
        • Radbruch L.
        • Brearley S.G.
        Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review.
        Palliat Med. 2017; 31: 684-706
        • Working Group on Priority Setting
        Priority setting for health research: Lessons from developing countries.
        Health Policy Plan. 2000; 15: 130-136
        • Viergever R.F.
        • Olifson S.
        • Ghaffar A.
        • Terry R.F.
        A checklist for health research priority setting: Nine common themes of good practice.
        Health Res Policy Syst. 2010; 8: 36
        • Alligier M.
        • Barres R.
        • Blaak E.E.
        • et al.
        OBEDIS Core Variables Project: European expert guidelines on a minimal core set of variables to include in randomized, controlled clinical trials of obesity interventions.
        Obes Facts. 2020; 13: 1-28
        • Hales S.
        • Lesher-Trevino A.
        • Ford N.
        • Maher D.
        • Ramsay A.
        • Tran N.
        Reporting guidelines for implementation and operational research.
        Bull WHO. 2016; 94: 58-64
        • Maclean P.S.
        • Rothman A.J.
        • Nicastro H.L.
        • et al.
        The Accumulating Data to Optimally Predict Obesity Treatment (ADOPT) core measures project: Rationale and approach.
        Obesity. 2018; 26: S6-S15
        • Pinnock H.
        • Barwick M.
        • Carpenter C.R.
        • et al.
        Standards for Reporting Implementation Studies (StaRI) statement.
        BMJ. 2017; 356: i6795
        • Rudan I.
        Setting health research priorities using the CHNRI method: IV. Key conceptual advances.
        J Globl Health. 2016; 6010501
        • Strifler L.
        • Cardoso R.
        • McGowan J.
        • et al.
        Scoping review identifies significant number of knowledge translation theories, models, and frameworks with limited use.
        J Clin Epidemiol. 2018; 100: 92-102
        • Black R.E.
        The legacy of the Child Health and Nutrition Research Initiative (CHNRI).
        J Glob Health. 2016; 6010101
        • International Consortium for Health Outcome Measurement. 2022
        https://www.ichom.org
        Date: 2022
        Date accessed: January 1, 2022
        • COMET Initiative
        Core Outcome Measures in Effectiveness Trials.
        http://www.comet-initiative.org
        Date: 2022
        Date accessed: January 1, 2022
        • Rosenbaum M.
        • Agurs-Collins T.
        • Bray M.S.
        • et al.
        Accumulating Data to Optimally Predict Obesity Treatment (ADOPT): Recommendations from the biological domain.
        Obesity. 2018; 26: S25-S34
        • Keeney S.
        • Hasson F.
        • McKenna H.
        Consulting the oracle: Ten lessons from using the Delphi technique in nursing research.
        J Adv Nurs. 2006; 53: 205-212
        • Public Health Agency of Canada
        Coronvirus disease (COVID-19).
        Government of Canada. 2022, Ottawa2020