NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Meta-Analytic Statistical Inferences for Continuous Measure Outcomes as a Function of Effect Size Metric and Other Assumptions
Methods Research Reports
Investigators: Blair T Johnson, PhD and Tania B Huedo-Medina, PhD.
Author Information and AffiliationsStructured Abstract
Introduction:
Meta-analysis cannot proceed unless each study outcome is on the same metric and has an appropriate sampling variance estimate, the inverse of which is used as the weight in meta-analytic statistics. When comparing treatments for trials that use the same continuous measures across studies, contemporary meta-analytic practice uses the unstandardized mean difference (UMD) to model the difference between the observed means (i.e., ME-MC) rather than representing effects in the standardized mean difference (SMD). A fundamental difference between the two strategies is that the UMD incorporates the observed variance of the measures as a component of the analytical weights (viz., sampling error or inverse variance) in statistically modeling the results for each study. In contrast, the SMD incorporates the measure’s variance directly in the effect size itself (i.e., SMD=[ME−MC]/SD) and not directly in the analytical weights. The UMD approach has been conventional even though its bias and efficiency are unknown; these have also not been compared with the SMD. Also unresolved is which of many possible available equations best optimize statistical modeling for the SMD in use with repeated measures designs (one or two groups).
Methods:
Monte Carlo simulations compared available equations in terms of their bias and efficiency across the many different conditions established by crossing: (1) number of studies in the meta-analysis (k = 10, 20, 50, and 100); (2) mean study sample sizes (5 values of N ranging from small to very large); (3) the ratio of the within-study observed measure variances for experimental and control groups and at pretest and post-test (ratios: 1:1, 2:1, and 4:1); (4) the post-test mean of each pseudo experimental group to achieve 3 parametric effect sizes (δ= 0.25, 0.50, and 0.80); (5) normal versus nonnormal distributions (4 levels); and (6) the between-studies variance (τ2= 0, 0.04, 0.08, 0.16, and 0.32). For the second issue, (7) the correlation between the two conditions was manipulated (ρpre-post = 0, 0.25, 0.50, and 0.75).
Results and Conclusions:
This investigation provides guidance for statistical practice in relation to meta-analysis of studies that compare two groups at one point in time, or that examine repeated measures for one or two groups. Simulations showed that neither standardized or unstandardized effect size indexes had an advantage in terms of bias or efficiency when distributions are normal, when there is no heterogeneity among effects, and when the observed variances of the experimental and control groups are equal. In contrast, when conditions deviate from these ideals, the SMD yields better statistical inferences than UMDs in terms of bias and efficiency. Under high skewness and kurtosis, neither metric has a marked advantage. In general, the standardized index presents the least bias under most conditions and is more efficient than the unstandardized index. Finally, the results comparing estimations of the SMD and its variance suggest that some are preferable to others under certain conditions. The current results imply that the choice of effect size metrics, estimators, and sampling variances can have substantial impact on statistical inferences even under such commonly observed circumstances as normal sampling distributions, large numbers of studies, and studies with large samples, and when effects exhibit heterogeneity. Although using the SMD may make clinical inferences more difficult, use of the SMD does permit inferences about effect size magnitude. The Discussion considers clinical interpretation of results using the SMD and addresses limitations of the current project.
Contents
- Preface
- Acknowledgments
- Peer Reviewers
- Introduction
- Methods
- Results
- Discussion
- References
- Glossary of Terms
- Appendix A Bias and Efficiency Results for Standardized and Raw Mean Differences (Specific Aim 1)
- Appendix B Bias, Efficiency, and Theoretical Variance for All Effect Size Indexes and Their Variances (Specific Aim 2)
Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services1, Contract No. 290-2007-10067-I, Prepared by: University of Connecticut–Hartford Hospital Evidence-Based Practice Center
Suggested citation:
Johnson BT, Huedo-Medina TB. Meta-Analytic Statistical Inferences for Continuous Measure Outcomes as a Function of Effect Size Metric and Other Assumptions. (Prepared by the University of Connecticut, Hartford Hospital Evidence-Based Practice Center under Contract No. 290-2007-10067-I.) AHRQ Publication No. 13-EHC075-EF. Rockville, MD: Agency for Healthcare Research and Quality; April 2013. www.effectivehealthcare.ahrq.gov/reports/final.cfm.
This report is based on research conducted by the University of Connecticut, Hartford Hospital Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2007-10067-I ). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
The information in this report is intended to help health care decision makers—patients and clinicians, health system leaders, and policymakers, among others—by improving the methods that meta-analyses use to accumulate data about health care services and other matters. This report is not intended to apply clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.
This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.
None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report.
- 1
540 Gaither Road, Rockville, MD 20850; www
.ahrq.gov
- NLM CatalogRelated NLM Catalog Entries
- Correction for bias in meta-analysis of little-replicated studies.[Methods Ecol Evol. 2018]Correction for bias in meta-analysis of little-replicated studies.Doncaster CP, Spake R. Methods Ecol Evol. 2018 Mar; 9(3):634-644. Epub 2017 Nov 21.
- The ratio of means method as an alternative to mean differences for analyzing continuous outcome variables in meta-analysis: a simulation study.[BMC Med Res Methodol. 2008]The ratio of means method as an alternative to mean differences for analyzing continuous outcome variables in meta-analysis: a simulation study.Friedrich JO, Adhikari NK, Beyene J. BMC Med Res Methodol. 2008 May 21; 8:32. Epub 2008 May 21.
- Heterogeneity estimation in meta-analysis of standardized mean differences when the distribution of random effects departs from normal: A Monte Carlo simulation study.[BMC Med Res Methodol. 2023]Heterogeneity estimation in meta-analysis of standardized mean differences when the distribution of random effects departs from normal: A Monte Carlo simulation study.Blázquez-Rincón D, Sánchez-Meca J, Botella J, Suero M. BMC Med Res Methodol. 2023 Jan 17; 23(1):19. Epub 2023 Jan 17.
- Review Psychological therapies for the treatment of post-traumatic stress disorder in children and adolescents (Review).[Evid Based Child Health. 2013]Review Psychological therapies for the treatment of post-traumatic stress disorder in children and adolescents (Review).Gillies D, Taylor F, Gray C, O'Brien L, D'Abrew N. Evid Based Child Health. 2013 May; 8(3):1004-116.
- Review Advantages of permutation (randomization) tests in clinical and experimental pharmacology and physiology.[Clin Exp Pharmacol Physiol. 1994]Review Advantages of permutation (randomization) tests in clinical and experimental pharmacology and physiology.Ludbrook J. Clin Exp Pharmacol Physiol. 1994 Sep; 21(9):673-86.
- Meta-Analytic Statistical Inferences for Continuous Measure Outcomes as a Functi...Meta-Analytic Statistical Inferences for Continuous Measure Outcomes as a Function of Effect Size Metric and Other Assumptions
Your browsing activity is empty.
Activity recording is turned off.
See more...