NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Methods for Evaluating Natural Experiments in Obesity: Systematic Evidence Review
Comparative Effectiveness Reviews, No. 204
Investigators: Wendy L Bennett, MD, MPH, Lawrence J Cheskin, MD, Renee F Wilson, MS, Allen Zhang, BS, Eva Tseng, MD, MPH, Oluwaseun Shogbesan, MD, Emily A Knapp, MHS, Elizabeth A Stuart, PhD, Eric B Bass, MD, MPH, and Hadi Kharrazi, MD.
Author Information and AffiliationsStructured Abstract
Objectives:
Obesity is an enormous public health problem among adults and children. Our objective was to systematically review studies evaluating programs and policies addressing obesity prevention and control in terms of their population-based data sources, use of data linkages, measures reported, study designs, and analytic approaches. The overarching goal of the review was to identify methodological advances that could strengthen research that uses natural experiments to evaluate the effectiveness of policies and programs to prevent and control obesity.
Data sources:
We systematically searched PubMed®, CINAHL®, PsycINFO®, and EconLit from 2000 to August 21, 2017, to identify all U.S. and non-U.S. studies of programs or policies targeting obesity prevention and control in people of all ages and in any setting.
Review methods:
Two independent reviewers screened abstracts and full-text articles. We required articles to be in English; address a program, policy, or built environment change; include 100 or more study subjects; and have a defined comparison or unexposed group. We used the Effective Public Health Practice Project (EPHPP) tool to rate studies for their risk of bias. This tool rates studies for their ability to draw causal inferences about program effectiveness.
Results:
The search identified 26,316 unique citations. Of the 294 studies (reported in 312 articles) eligible for inclusion (188 U.S. and 106 non-U.S.), 156 (53%) were natural experiment studies, 118 (40%) were experimental studies (randomized or nonrandomized controlled trials), and 20 (7%) had other study designs that did not fall into either of the other categories.
Data sources:
We identified 143 secondary data sources and 26 sharable primary data sources, totaling 116 sharable data sources after duplicates were removed. Criteria for a data system (data source exists, is available for research, is sharable, and contains outcomes of interest) were met by 106 data sources (71 U.S. and 35 non-U.S.). Sixty-two percent of the U.S. data systems contained at least one of the main measures for weight or body mass index in adults or children, or dietary or physical activity behaviors. Fifty-three percent of the U.S. data systems included at least one outcome related to the food environment, physical activity environment, commuting behavior, or purchasing behavior, or included information about a relevant exposure in a policy, program, or built environment change. These 71 U.S. data systems often reported more than one outcome. Thirty-seven percent of the U.S. data systems were linked with a secondary data source or system other than the primary data source. Most studies that linked their data systems with external data systems used an individual-level key or a geographic allocation.
Outcomes/measures:
Of the 294 included studies, we identified 112 studies with childhood weight measures, 32 studies with adult weight measures, 152 studies with physical activity measures, and 148 studies with dietary measures. Thirty-seven of the 294 studies reported on outcomes related to the food environment, physical activity environment, commuting behavior, or purchasing behavior.
Study design and methods:
Natural experiment studies most commonly used cross-sectional comparisons of exposed and unexposed groups (n=55; 35%). Difference-in-differences approaches that compared exposed and unexposed groups before and after an exposure were used in 45 studies (29%), while 48 studies (31%) used pre/post designs that compared one group before and after an exposure. Most natural experiment studies were rated as having a “weak” global rating (i.e., high risk of bias), with 63 percent having a weak rating for handling of withdrawals and dropouts, 42 percent having a weak rating for study design, 40 percent having a weak rating for confounding, and 26 percent having a weak rating for data collection. Experimental studies were rated as “strong” (low risk of bias) in study design, control of confounding, and data collection methods, but were weaker in blinding and selection bias. We identified methodological and analytic advances that would help to strengthen efforts to estimate the effect of programs, policies, or built environment changes on obesity prevention and control, such as consistent use of data dictionaries, reporting standards on linkage methods of data sources, data sources with long-term public health surveillance of obesity and health behavioral outcomes, and use of study designs with multiple pre- and post-exposure time points.
Conclusions:
Our systematic review identified numerous natural experiment studies (n=156) and data sources, including sharable and non-sharable data sources (n=216), that have been used to estimate the effect of programs, policies, or built environment changes on obesity prevention and control. The studies used a wide variety of outcome measures and analytic methods, often with substantial risk of bias. The findings reinforce the need for methodological and analytic advances that would strengthen efforts to improve obesity prevention and control.
Contents
- Purpose of Review
- Key Messages
- Preface
- Acknowledgments
- Technical Expert Panel
- Peer Reviewers
- Introduction
- General Methods and Results
- Key Question 1. Population-Based Data Sources
- Key Question 2. Linking Data Sources
- Key Question 3. Assessment Measures
- Key Question 4. Experimental and Non-Experimental Methods
- Key Question 5. Risk of Bias
- Key Question 6. Methodological Advances
- Discussion
- References
- Appendix A. Acronyms
- Appendix B. Glossary of Key Terms
- Appendix C. Detailed Search Strategies
- Appendix D. Abstraction Forms
- Appendix E. Data System Classification/Coding Schema
- Appendix F. Approach to Identifying Methodological/Analytic Advances That Would Help Strengthen Efforts To Estimate the Effect of Programs, Policies, or Built Environment Changes on Obesity Prevention and Control
- Appendix G. Included Studies
- Appendix H. Evidence Tables
- Appendix I. Programs, Policies, or Built Environment Changes in Included Studies (U.S. Only)
- Appendix J. Data Systems Identified by the Systematic Review
- Appendix K. Individual Study Risk of Bias Ratings
- Appendix L. Comparison of Two Tools for Assessing Study Risk of Bias
Suggested citation:
Bennett WL, Cheskin LJ, Wilson RF, Zhang A, Tseng E, Shogbesan O, Knapp EA, Stuart EA, Bass EB, Kharazzi H. Methods for Evaluating Natural Experiments in Obesity: Systematic Evidence Review. Comparative Effectiveness Review No. 204. (Prepared by the Johns Hopkins University Evidence-based Practice Center under Contract No. 290-2012-00007-I.) AHRQ Publication No. 18-EHC006-EF. Rockville, MD: Agency for Healthcare Research and Quality; December 2017. Posted final reports are located on the Effective Health Care Program search page. DOI: http://doi.org/10.23970/AHRQEPCCER204.
This report is based on research conducted by the Johns Hopkins University Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2012-00007-I). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report.
The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well-informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.
AHRQ or U.S. Department of Health and Human Services endorsement of any derivative products that may be developed from this report, such as clinical practice guidelines, other quality enhancement tools, or reimbursement or coverage policies, may not be stated or implied.
This report may periodically be assessed for the currency of conclusions. If an assessment is done, the resulting surveillance report describing the methodology and findings will be found on the Effective Health Care Program Web site at www.effectivehealthcare.ahrq.gov. Search on the title of the report.
- 1
5600 Fishers Lane, Rockville, MD 20857; www
.ahrq.gov
- NLM CatalogRelated NLM Catalog Entries
- Methods for Evaluating Natural Experiments in Obesity: A Systematic Review.[Ann Intern Med. 2018]Methods for Evaluating Natural Experiments in Obesity: A Systematic Review.Bennett WL, Wilson RF, Zhang A, Tseng E, Knapp EA, Kharrazi H, Stuart EA, Shogbesan O, Bass EB, Cheskin LJ. Ann Intern Med. 2018 Jun 5; 168(11):791-800. Epub 2018 May 1.
- Review Screening and Interventions for Childhood Overweight[ 2005]Review Screening and Interventions for Childhood OverweightWhitlock EP, Williams SB, Gold R, Smith P, Shipman S. 2005 Jul
- Review Behavioral Counseling to Promote a Healthful Diet and Physical Activity for Cardiovascular Disease Prevention in Adults Without Known Cardiovascular Disease Risk Factors: Updated Systematic Review for the U.S. Preventive Services Task Force[ 2017]Review Behavioral Counseling to Promote a Healthful Diet and Physical Activity for Cardiovascular Disease Prevention in Adults Without Known Cardiovascular Disease Risk Factors: Updated Systematic Review for the U.S. Preventive Services Task ForcePatnode CD, Evans CV, Senger CA, Redmond N, Lin JS. 2017 Jul
- Review Screening for Adolescent Idiopathic Scoliosis: A Systematic Evidence Review for the U.S. Preventive Services Task Force[ 2018]Review Screening for Adolescent Idiopathic Scoliosis: A Systematic Evidence Review for the U.S. Preventive Services Task ForceDunn J, Henrikson NB, Morrison CC, Nguyen M, Blasi PR, Lin JS. 2018 Jan
- Review Lipid Screening in Childhood and Adolescence for Detection of Familial Hypercholesterolemia: A Systematic Evidence Review for the U.S. Preventive Services Task Force[ 2016]Review Lipid Screening in Childhood and Adolescence for Detection of Familial Hypercholesterolemia: A Systematic Evidence Review for the U.S. Preventive Services Task ForceLozano P, Henrikson NB, Dunn J, Morrison CC, Nguyen M, Blasi PR, Anderson ML, Whitlock E. 2016 Aug
- Methods for Evaluating Natural Experiments in Obesity: Systematic Evidence Revie...Methods for Evaluating Natural Experiments in Obesity: Systematic Evidence Review
Your browsing activity is empty.
Activity recording is turned off.
See more...