Criteria for Distinguishing Effectiveness From Efficacy Trials in Systematic Reviews
Technical Reviews, No. 12
Authors
Investigators: Gerald Gartlehner, MD, MPH, Richard A Hansen, PhD, Daniel Nissman, MD, MPH, Kathleen N Lohr, PhD, and Timothy S Carey, MD, MPH.Affiliations
Structured Abstract
Objectives:
To propose and test a simple instrument based on seven criteria of study design to distinguish effectiveness (pragmatic) from efficacy (explanatory) trials while conducting systematic reviews.
Design:
Currently, no validated definition of effectiveness studies exists. We asked the directors of 12 Evidence-based Practice Centers (EPCs) to select six studies each: four that they considered to be examples of effectiveness trials and two considered efficacy studies. We then applied our proposed criteria to test the construct validity using the selected studies as if they had been identified by a gold standard.
Results:
Based on the rationale to identify effectiveness studies reliably with minimal false positives (i.e., a high specificity), a cut-off of six criteria produced the most desirable balance between sensitivity and specificity. This setting produced a specificity of 0.83 and a sensitivity of 0.72.
Conclusions:
When applied in a standardized manner, our proposed criteria can provide a valid and simple tool to distinguish effectiveness from efficacy studies. The applicability of systematic reviews can improve when analysts place more emphasis on the generalizability of included studies. In addition, clinicians can also use our criteria to determine the external validity of individual studies given an appropriate population of interest.
Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services.1 Contract No. 290-02-0016. Prepared by: RTI-International-University of North Carolina Evidence-based Practice Center, Research Triangle Park, NC.
Suggested citation:
Gartlehner G, Hansen RA, Nissman D, Lohr KN, Carey TS. Criteria for Distinguishing Effectiveness From Efficacy Trials in Systematic Reviews. Technical Review 12 (Prepared by the RTI-International-University of North Carolina Evidence-based Practice Center under Contract No. 290-02-0016.) AHRQ Publication No. 06-0046. Rockville, MD: Agency for Healthcare Research and Quality. April 2006.
This report is based on research conducted by the RTI-International-University of North Carolina Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-02-0016). The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
The information in this report is intended to help clinicians, employers, policymakers, and others make informed decisions about the provision of health care services. This report is intended as a reference and not as a substitute for clinical judgment.
This report may be used, in whole or in part, as the basis for the development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.
- 1
540 Gaither Road, Rockville, MD 20850. www
.ahrq.gov