Identify Staffing

The actual work and associated decisions of conducting the systematic review are the responsibility of a multidisciplinary research team. However, at appropriate times in the review process, it is also desirable to solicit input from external experts, sponsors and users. The process of obtaining external inputs needs to be defined before starting the project to ensure independence of the review from vested interests and potentially biased perspectives while ensuring that the research team has the information needed to achieve subject matter appropriateness and usefulness of the review.

Form Multidisciplinary Research Team

Once the topic has been defined, the initial step in starting the systematic review process is to form a multidisciplinary research team. The research team is responsible for all of the activities and decisions involved in the conduct of the systematic review and must be free of actual or apparent biases relative to the particular topic area under review. The research team should include systematic review methodologists. In addition, depending on the nature of the topic and how the results will be used, the research team will generally also include, but not be limited to, domain experts (e.g., nutrition scientists), clinicians, epidemiologists, and statisticians. In forming the research team for a nutrition-related topic, it is important to include nutrition scientists and at least one scientist with a wide rather than narrow range of views and expertise on the topic under review. A broadly-based research team works together to identify search terms, develop an analytic framework, answer technical questions, clarify relationships among related topics and provide input during the peer review process.

Plan for Outside Inputs

Outside inputs can enhance the quality and usefulness of the review. However, these inputs need to be carefully managed to avoid the potential introduction, or appearance, of bias and vested interests into the review process. Ideally, this is achieved through a prior definition of the roles and responsibilities of the multidisciplinary research team relative to the outside inputs. In all cases, the outside inputs are advisory in nature with the ultimate decisions related to the conduct and decisions involved in the review solely in the hands of the research team. In those cases where a review project has identified sponsors and/or users, an early consultation among the research team and the sponsors or users to ensure a common understanding of the scope of work and user needs can help to ensure the usefulness of the review. Specific subject matter experts and/or an advisory committee representing a wide range of expertise that also often includes persons with varying perspectives may be convened to provide comment on the analytic framework, research questions, eligibility criteria and search terms. Finally, the rigor of the review can be enhanced by the use of external peer reviewers for the final draft review.

Develop Analytic Framework

An analytic framework assists in the synthesis and interpretation of the study results and in some cases serves as a guide for the integration of information from multiple types of data. In general, the framework is developed by the systematic review by a collaborative effort of the domain experts and the methodologists, and reviewed and refined by other members of the research team. The analytic framework is used by the systematic review methodologists as they review and summarize the data. It has been used successfully by the U.S. Preventive Services Task Force for many years to help formulate research questions.⁽²⁰⁾ Analytic frameworks provide visual maps outlining specific linkages among the populations of interest, exposures, modifying factors, biological role of a nutrient and outcomes of interest. These frameworks depict the chain of logic that evidence must support to link the exposure to clinical outcomes and should be identified a priori. Defining these relationships can be helpful in further refining the key questions and study eligibility criteria prior to starting the literature search, and in interpreting relevant studies once they are identified. In the case of nutrition, the analytical framework reflects the known biological mechanisms of the nutrient and guides in integrating the various types of information available into a coherent picture. An example of the analytic framework used for a systematic review addressing the area of (n-3) fatty acids and cardiovascular disease is provided ( Figure 1 ).

Figure

Figure 1. Analytic framework for (n-3) fatty acid exposure and cardiovascular disease. This framework concerns the effect of (n-3) fatty acid exposure (as a supplement or from food sources) on cardiovascular disease. Populations (more...)

Develop and Refine Research Questions

Developing and refining the research question(s) is a collaborative effort between the research team and, when appropriate, sponsors and intended users of the systematic review. Frequently, there is an overarching question that needs to be broken down to smaller questions that can be addressed. Well formulated question(s) are critical in ensuring that the systematic review will be useful in addressing the intended goals and needs of the project. The question(s) define the scope of the project, determine the search terms, inform the literature selection and evaluation, and dictate the approach to data synthesis. The types of key questions can vary widely depending on the purpose of the systematic review. Multiple questions are typically needed to address even narrowly defined topics, which are subsequently combined to form conclusions. The diverse types of questions developed for the example reviews reflecting both the sponsor interests and the available literature are presented (Table 2).

The ‘PICO’ approach is commonly used to formulate research questions. The acronym PICO stands for Population (Participants), Intervention (or Exposure for observational studies), Comparator and Outcomes.⁽²²⁾ Thoughtfully and unambiguously specifying the parameters for each of these attributes allows for research questions to be created that will yield the intended outcome. Various combinations of these parameters form potentially useful questions. In formulating each question, it is necessary to consider the tradeoffs between the desire for ideal knowledge and the reality of limited data, study designs and available resources. An example of a question and component parts is presented, ‘What is the overall 5 y mortality in various populations of taking 1 g of fish oil daily compared with those taking a placebo?’ (Table 3). Alternately, a different question can be generated by selecting an entry from each of the components of the PICO approach (columns of the table) and applying modifiers of interest. For example, ‘What is the 5 y overall mortality in general populations taking 1 g of fish oil supplement daily compared with those taking a isocaloric fat placebo?’.

Table 3

Example of the PICO method of formulating a systematic review question¹.

Define Eligibility Criteria

The PICO components define much of the eligibility criteria for selecting the studies. Additional criteria include study design, minimum/maximum dose levels (plausibility at dietary or pharmacological level), minimum number of subjects per study arm, background diets, baseline nutritional status, minimum intervention period, minimum information for characterizing the intervention (placebo, active intervention), outcome measures of interest and statistical analysis. Additional topic specific criteria are often necessary. In the rare instances where many more potentially relevant articles may be available than feasibly can be reviewed within resources and time available, one might limit the review of the literature to larger and more recent studies. It is important that these decisions be made in consultation with domain experts knowledgeable about the topic of interest. In some cases limiting the review to, for example, more recent studies can result in the loss of unique data that due to resources, ethics or other reasons have not been duplicated recently. Examples of eligibility criteria for the three example reviews show the diverse types of data used to answer the range of questions that reflect different interests and needs of the sponsors of these systematic reviews (Table 2).

Identify Search Terms

The list of search terms, developed my the multidisciplinary team, must be adequate in scope to capture all of the relevant literature but narrow enough to avoid capturing so much extraneous literature that an undue burden is placed on the research team. To be comprehensive, multiple databases (e.g., MEDLINE^®, CAB Abstracts, and Cochrane Library Central) as well as citations of relevant retrieved articles should be searched, supplemented by contributions of domain experts. The number of key search terms used in the 3 example reviews ranged from 33 to 130 (Table 2).

Perform Literature Search

At this point the domain experts step back from the review process and the methodologists conduct the literature search and summarize the findings. This division of labor ensures a level of objectivity unencumbered by potential biases of domain experts. Clear documentation of the search strategy used and bibliographic databases searched is an inherent part of a systematic review. It facilitates the ability of other groups to reproduce the systematic reviews, allows comparisons across reviews so users can assess their similarities and differences, and serves as a foundation for an efficient updating of the systematic reviews as new findings emerge. In addition, this documentation also facilitates other uses of a systematic review by clarifying both its breath and boundaries.

Evaluate Search Results

Systematic reviews of nutrition topics typically evaluate a diverse body of literature that can be diffuse and voluminous. For this reason, screening abstracts guided by eligibility criteria for potentially relevant articles in a consistent, comprehensive and efficient manner is critical to the integrity of the systematic review. Once potentially relevant literature is identified, full-text articles are retrieved and reviewed for inclusion on the basis of the predetermined criteria. For one topic, Effects of Soy on Health Outcomes, ⁽¹²⁾ the initial literature search yielded about 4800 citations (Table 2). Five hundred ninety-nine potentially relevant full-text articles were retrieved for further evaluation. One hundred seventy-eight articles met inclusion criteria and were included in the final report. A flow diagram depicting the process of literature evaluation and a rejection log of retrieved full-text articles along with the reasons for exclusion should be provided to enhance transparency.

Construct Evidence and Summary Tables, and Extract Data

Data need to be extracted which will identify information that is important in evaluating the quality and relevance of a study using nutrient-specific criteria in addition to those criteria commonly used. Nutrient supplement information might include intake/dose, source of supplement and chemical analysis, chemical form, mode of delivery, route and duration of delivery, and measures of prior nutritional status. Additional types of information might include level of the nutrient in the background diet, method used to estimate intake, analytical methods used to assess nutrient status and whether a nutrient biomarker or other approach was used to validate the dietary data. An evidence table is a comprehensive compilation of a priori defined data elements extracted from the primary studies that are judged to be important in the interpretation of the evidence. A summary table is a distillation and synthesis of information from evidence tables. It is typically used to succinctly present study characteristics and results in a report or manuscript to support the interpretation of the evidence addressing a specific question. While a study will usually be found only once in evidence tables, the same study may appear in multiple summary tables addressing different questions. Construction of evidence and summary tables is critical to ensuring that all relevant data are extracted and tabulated in a format that will lead itself to subsequent uses. The actual extraction, depending on the nature of that available, may involve data derived from different types of study designs (observation studies, randomized controlled trials, and animal and in vitro studies). Consistent with the different study designs, the format of evidence and summary tables can be adapted to accommodate the types of relevant information important to extract from the full-text articles. The type of acceptable study design and needed information to be included in evidence and summaries tables must be specified a priori.

Assess Methodological Quality and Applicability of Studies

Studies included in a systematic review have different protocols, are conducted with different levels of rigor and their results reported in a variety of manners. These variations may be manifested as discrepancies of results across studies. Thus it is important to assess studies for potential bias due to methodological deficiencies and to assess how variations of study conduct (e.g., population enrollment) may influence the results. Critical appraisal of the studies helps to interpret the effects of methodological and clinical/biological heterogeneity on the results. Certain features of study design and conduct such as randomization and blinding in randomized controlled trials, when poorly executed, could result in biased estimates. The effect of these factors, however, is difficult to predict in a specific study. ⁽²³⁾ Thus, while critical appraisal of studies is guided by certain principles, there are some inevitable subjective components that reviewers and readers should be aware of. Numerous approaches to appraise evidence have been proposed emphasizing different aspects of study design, conduct and reporting.⁽²⁴⁾ An example of the assessment of methodological quality and applicability of individual studies is depicted (Table 4). The Cochrane Collaboration ⁽²⁵⁾ and the U.S. Preventive Services Task Force ⁽²⁶⁾ and an international group, the Grading Recommendations Assessment, Development and Evaluation [GRADE] working group ⁽²⁷⁾ propose a next step which is to rate the overall strength of the body of evidence. This step integrates an estimation of the overall risk of bias of evidence based on methodological study quality as described above with estimations of the directness, consistency, and precision of the evidence. Rating the applicability of the evidence to the target population is also done at this step. The applicability of these approaches to nutrition has not as yet been evaluated.

Table 4

One approach to assessing methodological quality and applicability of studies.

Perform Meta-analysis, as Appropriate

Meta-analysis uses statistical methods to combine two or more studies addressing the same question. It is often part of a systematic review and can identify significant results when individual studies are inadequately powered. Most meta-analyses combine results across studies in order to arrive at an overall estimate. When data are available, meta-regression can be performed to explain discrepancies across studies and to explore variations of effects such as dose response relationships. Sometimes meta-analyses may shed new insights that studies examined individually may fail to reveal. ⁽²⁸⁾ Statistical methods to perform meta-analyses have advanced in the past two decades and the strengths and limitations are well understood. A key issue in performing a meta-analysis is the appropriateness of combining studies. This decision should be weighted in the context of the nature of the data and how the results will be used. Because several meta-analyses addressing similar questions may result in dissimilar conclusions due to differences, at times small, in the questions asked, the inclusion criteria applied, and the method of assessing methodological quality and applicability of studies used, it is important in interpreting the results to carefully understand the questions and eligibility criteria.

Synthesize Results

It bears remembering that answers obtained from systematic reviews address only the identified questions. Users of the systematic review (e.g., government agencies, expert panels) must then integrate results from the systematic review with other information to form their practice recommendations or public policies. Sometimes a systematic review may find no or only poor quality evidence or identify inconsistencies among study results. These data would suggest areas where future research needs to be conducted.