1.A. Healthcare Cost and Utilization Project (HCUP) QIs

The demand for information on quality in health care has risen sharply over the past decade. In response to this demand, the Healthcare Cost and Utilization Project (HCUP I) Quality Indicators (QIs) were developed at the Agency for Healthcare Research and Quality (AHRQ, formerly Agency for Health Care Research and Policy) in 1994. HCUP is a federal-state-private collaboration to build uniform databases from administrative hospital data collected by state data organizations and hospital associations. These data are gathered by AHRQ, converted into a uniform format, and released for research purposes. Currently, 22 states participate in HCUP. In the early 1990s, the HCUP state partners requested assistance in making greater use of the administrative data they collected. In response, AHRQ researchers conducted a literature review to identify measures of quality that could be constructed using only information routinely found on hospital discharge abstracts and claims. The HCUP I Quality Indicator software was developed and disseminated.

The 1994 HCUP indicator set (HCUP I), which is in use currently, consists of 33 indicators constructed using administrative data available in the Nationwide Inpatient Sample (NIS), one of the HCUP databases. Included in the set are indicators of procedure utilization, ambulatory care sensitive condition admissions, post-operative and other complications of care, and mortality. Developers of this indicator set compiled a list of quality indicators used by a variety of organizations, and then selected indicators based on simplicity, feasibility, evidence from a review of the literature, and coverage of a range of potentially affected patient populations. The 33 resulting quality indicators were designed to highlight quality concerns, particularly in clinical performance, and to target areas for further analysis. As a result, the indicators were not intended as definitive measures of quality problems, but rather as screens for use in quality improvement. As screening tools, these indicators would serve as a first-round flag of potential quality problems, which should be investigated further by other methods, such as chart review.

Since indicators with complex multivariate adjustments can be difficult to understand and to interpret, the original HCUP I indicators were developed without such techniques, and with the objective of simplicity. Two approaches were utilized to control for differences in severity of illness or case mix across hospitals. First, numerators and denominators defined to assess rates for each indicator were restricted to homogenous groups whenever feasible. For example, QIs examining in-hospital mortality following common elective procedures restrict the populations at risk to uncomplicated cases only. The second approach to controlling for differences in case mix employed standardization based on the Major Diagnostic Code (MDC) mix of the hospital compared to the state's MDC mix.³

HCUP I QIs were released publicly in 1994 as a set of statistical programs. ⁴ AHRQ has disseminated widely this software tool in both SAS and SPSS statistical package formats. Current users of these indicators include hospital associations, state health departments, data organizations, and individual hospitals. In general, these indicators are used for preliminary quality screening, internal reporting, and, in a few cases, non-public benchmarking. Specific examples of use include:

1. hospitals have refined their patient selection criteria for procedures to improve appropriate usage;
2. hospitals have reviewed medical records to verify the presence of poor outcomes and to investigate the reasons behind the event identified by the QIs;
3. the QIs have served as a springboard for hospitals to identify community health care needs and plan for particular services (e.g., one hospital worked with the surrounding community to strengthen outpatient services for diabetes patients after seeing QI data);
4. a project with one state chapter of the American College of Obstetricians Gynecologists was initiated to reduce adverse outcomes associated with low birth weight infants; and
5. use of the QIs has provided hospitals the opportunity to verify and improve the accuracy of the data they reported.

Although the original 33 HCUP I QIs were designed using a multi-state hospital discharge data set, they can provide information on care both within and outside hospitals. Some of the QIs offer hospitals self-assessment opportunities in the areas of potentially avoidable adverse hospital outcomes and of potentially inappropriate utilization of hospital procedures. Other QIs cover potentially avoidable hospital admissions, and therefore provide public health officials insight into the quality and appropriateness of medical care in the community.

1.B. Evidence Project to Improve HCUP QIs

Since the time of the original HCUP I development work in 1994, numerous managed care organizations, state Medicaid agencies and hospital associations, quality improvement organizations, the National Committee on Quality Assurance (NCQA), the Joint Commission on the Accreditation of Healthcare Organizations (JCAHO), academic researchers and others have contributed substantially to the knowledge base of hospital quality indicators. Based on input from current users and on advances to the scientific base for specific indicators, AHRQ decided to fund a research project to refine and further develop the HCUP QIs. As a result, AHRQ charged the UCSF-Stanford Evidence-based Practice Center (EPC) to revisit the initial 33 indicator set (HCUP I QIs), evaluate their effectiveness as indicators, identify potential new indicators, and ultimately propose a revised set of indicators. This report documents the evidence project to develop recommendations for improvements to the HCUP I indicators.

Three primary goals were established to accomplish this task: 1) Identify potential new indicators, 2) Evaluate existing HCUP I indicators and potential indicators using both literature review and empirical analyses of indicator performance, and 3) Examine the need for risk adjustment of recommended indicators. As part of the indicator identification goal, AHRQ requested that additional QIs cover populations and conditions that are not well represented in the HCUP I QIs. In particular, the EPC was expected to review the evidence for indicators in four priority expansion areas: 1) Pediatric conditions, 2) Medical conditions including chronic illnesses, 3) Potentially avoidable hospitalizations, particularly in Healthy People 2010 objective areas, ⁵ and 4) New procedures not covered in the HCUP I QIs. For the indicator evaluation goal, AHRQ requested that the current QIs be reviewed for their appropriateness in terms of choice of denominator (discharge-based versus population-based) and for their overall validity as a measure of quality. For the final goal, the EPC was asked to evaluate more sophisticated methods of risk adjustment using hospital discharge data, based on input from experts in the field, and literature reviews.

The team designed a series of investigations to accomplish these goals. These included telephone interviews of those knowledgeable about quality measurement, two phases of extensive literature reviews, and a series of empirical analyses using the HCUP data sets. The in-depth review, supplemented by extensive empirical evaluation, focused on information that would be useful for implementing a revised set of HCUP quality indicators, hereafter referred to as the HCUP II QIs.

Six specific key questions were formulated to guide the research process:

• What indicators are currently in use or described in the literature that could be defined using HCUP discharge data?
• What are the quality relationships reported in the literature that could be used to define new indicators using HCUP discharge data?
• What evidence exists for indicators in AHRQ's designated expansion areas - pediatric conditions, chronic disease, new technologies, and ambulatory care sensitive conditions?
• Of the existing HCUP I and potential indicators, which ones have literature-based evidence to support face validity, precision of measurement, minimum bias, and construct validity of the indicator?
• Given the potential importance of risk adjustment for appropriate interpretation of the indicators, what risk-adjustment method should be supported, given the limits of administrative data and other practical concerns, for use in conjunction with the recommended indicators?
• Of the existing HCUP I and potential indicators, which ones perform well on empirical tests of precision of measurement, minimum bias, and construct validity?

1.C. Anticipated Uses of this Evidence Report

The approach to identification and evaluation of QIs presented in this report underpins the development of the HCUP II QIs. The primary goal of the report is to document the evidence, both from the literature and from empirical analysis, on quality indicators suitable for use based on hospital discharge abstract data. By identifying and evaluating potential indicators and risk adjustment strategies, the report may serve as a springboard for commentary on proposed recommendations for specific improvements to the HCUP I QIs.

This report is likely also to serve as a reference for background material on quality measurement, and as a review of the current state of quality indicators and risk adjustment methods. In addition, it documents a novel application of evidence-based methods to quality indicator evaluation, adapting the traditional use of these methods for informing clinical practice choices.

We anticipate that the audience for this report is likely to be quite heterogeneous, including health care providers, organizations, policy makers, and public health officials. A primary audience for this report is the AHRQ along with those using the current HCUP I QI tool since the report provides details about the methods and evaluation results for the recommended indicators. However, the report has been written for a wider audience likely to be interested in improvements to quality measurement in general, and the HCUP QI set specifically. Indicators applicable to health care providers and those actionable by public health officials were both considered in the systematic review and evaluation of potential new indicators for the HCUP II QI set. Therefore, it is likely that each audience will find differing aspects of the report most useful, and as such, we have included a variety of information.

The HCUP II QIs themselves should provide a means of using a readily available data source, hospital administrative data, to provide initial screening of quality of care. These measures can be used to provide national and regional benchmarks against which individual providers, localities, and states can compare themselves. In addition, with an understanding of the limitations of administrative data and with appropriate precautions, the QIs can form a preliminary basis of a quality improvement program. Because of the limitations associated with administrative data, the HCUP QIs are not designed for purchasing decisions and public reporting on individual hospitals.

Potential stakeholders and quality measurement experts were selected to review the report (see Appendix 1 for a list of reviewers). After receiving reviewer comments, the project team revised the report and sent it to an expanded group for a second round of review (also listed in Appendix 1), followed by further revisions. As a result, this report has undergone extensive peer review and subsequent revision based on reviewer comments.