Our review of the literature and empirical performance of potential HCUP II quality indicators suggested that HCUP data can play a useful role in policy and management activities to improve quality, and might play an even more extensive role in the future. For quality indicators based on volume of inpatient procedures, HCUP is an excellent data source. For some other specific quality indicators related to the utilization of inpatient services, HCUP data can also provide some important insights. For example, because the "appropriate" rates of procedures like incidental appendectomy and bilateral cardiac catheterization are low, high hospital rates on these recommended HCUP II indicators are very likely to represent inappropriate care. For a range of other hospital quality indicators related to important outcomes - mortality for inpatient conditions and inpatient procedures - HCUP data reveal substantial differences across hospitals. If the high-rate hospitals could achievethe mortality rates of the lower-rate hospitals, literally thousands of inpatient deaths per year might be avoided. Our review also identified some potentially quite valuable uses of HCUP data for detecting important differences in the quality of outpatient care, as reflected in inpatient admissions for a range of chronic conditions that can usually be managed effectively on an ambulatory basis, and for acute complications that can be prevented through good ambulatory care.

Despite the promising findings of our review, we also identified some important limitations of indicators that could be constructed from HCUP data. Some of these limitations could be addressed through additional action and research. The limitations include:

1. Some indicators, such as hospital volume indicators, are only proxies for quality. For example, while on average hospitals with higher volumes of certain complex procedures have been shown to have better outcomes, procedure volume is only weakly correlated with mortality and other outcomes, and volume indicators may encourage inappropriate utilization. It is important that the volume-outcome relationship or other relationships on which proxy measures are based be revisited to assure the validity of these indicators.
2. A few indicators, such as laparoscopic cholecystectomy and congestive heart failure, may be susceptible to selection bias because the cases ascertainable from HCUP data do not represent the universe of patients with that condition or procedure. Injudicious use of these indicators may lead health care providers to admit patients who do not actually require inpatient care. A related problem is that inadequate or variable coding of key diagnoses may interfere with ascertainment of cases, such as for vaginal births after cesarean delivery.
3. Missing information about patient outcomes is a potential source of bias for several mortality indicators, because 30-day mortality for some conditions substantially exceeds inpatient mortality. As a result, injudicious use of these indicators may lead health care providers to discharge patients prematurely, thereby shifting deaths to the outpatient setting.
4. Patient characteristics, such as disease severity, comorbidities, physiologic derangements, and functional status, may substantially affect performance on mortality measures, and may vary systematically across providers or areas. In particular, confounders that cannot be identified from HCUP data, such as clinical information derived from physical examination, laboratory tests, and radiographic findings, and functional abnormalities, are of concern. Injudicious use of these indicators may lead health care providers to avoid high-risk patients, whose acuity is not fully captured by APR-DRGs or other severity classification systems, or to "upcode" comorbidities.
5. Many indicators have limited evidence of construct validity, meaning that poor performance has not been clearly associated with poor quality of care (according to generally accepted process or risk-adjusted outcome indicators). Ideally, such associations should be demonstrated at both the patient and hospital or area levels. Without such evidence, providers or areas may find it difficult to address performance problems and identify opportunities for process improvement.
6. Empirical analyses demonstrated that many indicators are somewhat imprecise, meaning that there is significant random variation making it difficult to discern substantial systematic effects at the provider level. The recommended indicators all demonstrated sufficient precision for use as quality indicators, but some benefit substantially from smoothing techniques. Using indicators without such techniques may lead to inappropriate conclusions about provider performance.

Given these limitations, quality indicators derived from currently available HCUP data are best used as "quality screens," meaning that they are particularly useful in identifying potential quality problems and identify areas for further investigation. These indicators may serve additional functions, as long as they are interpreted with consideration of the caveats noted in this report. For example, hospitals, hospital systems, hospital associations, local, state and Federal health agencies, managed care organizations, purchasers or consumer coalitions may use these indicators to compare health system performance with regional, national, or international benchmarks. Hospitals, hospital systems, or other provider representatives may use these indicators to reallocate resources among facilities or departments, or to develop and evaluate performance improvement activities. Regardless of the use of the QIs it is essential that the limitations of these measures be addressed.

Our principal findings suggest some directions for further research and development of HCUP and similar discharge-based data systems. To permit more confident conclusions based on HCUP II measures - that is, to improve the predictive power of HCUP II-based screens - additional research should be undertaken to assess the validity of promising measures. Many studies reviewed in this report have done this already, and have provided valuable evidence on the performance of discharge-based quality indicators. However, the previous studies have had a number of limitations. First, many studies focused on the validity of quality measures at the patient level, or whether patients who screen positive on the measure are more likely to have received poor quality care than patients who screen negative. But because these studies typically included only one or a few hospitals, they were unable to address a potentially more important question: at the level of hospitals or areas, does poor performance on an HCUP II indicator reflect a true quality problem? In other words, do hospitals with poorer performance on an HCUP II indicator provide lower quality of care than those with better performance on the same indicator? Such studies would require combining detailed process-of-care data from multiple hospitals with quality indicators based on administrative data (ideally with smoothing methods and risk adjustment) for the same group of hospitals and patient admissions. Although the cost of these investigations might seem prohibitive, there are efforts proposed, underway, and even completed to collect clinically-detailed data for a large and reasonably representative set of hospitals. For example, HCFA's Cooperative Cardiovascular Project (CCP) collected comprehensive chart review data for all heart attack patients in the traditional Medicare program admitted to virtually all hospitals in the United States during a 9-month period in 1994-95. Thus, at relatively low cost, it would be possible to extend recent work comparing hospital performance in heart attack care based on detailed CCP process measures to hospital performance based on recommended HCUP II indicators for the same patients. Such studies would not only be useful for assessing the performance of HCUP II measures, but also for identifying where supplemental clinical data collection is most needed to evaluate quality of care.

Second, more research is needed to assess whether and how alternative risk adjustment methods affect indicator performance. Once again, many useful studies reviewed in our report have already been performed. Often, they found that a hospital's measured performance was not only sensitive to whether risk adjustment was performed, but also to which risk adjustment system was used. However, such studies have not been performed for most of the recommended indicators. Moreover, these studies usually did not take account of the improving data now available, including up to 25 fields for secondary diagnoses and flag variables (in New York and California) that distinguish between comorbidities present on admission (which generally should be included in risk adjustment) and complications that develop after admission (which generally should not). In addition, risk adjustment methods for administrative data are improving, which will benefit both quality measures and reimbursement systems. Finally, the previous studies generally did not account for random variation across hospitals in comparing different risk adjustment methods. Because such random variation contributes substantially to the apparent variation across hospitals for most HCUP II quality indicators, it is not surprising that relationships across risk adjustment systems are often weak. Simply put, the problem of serious "measurement error" complicates the detection of significant, consistent relationships. Future research using newer HCUP data, including the software accompanying this report that generates smoothed estimates, could address these limitations.

Even if these limitations of previous research are addressed in future studies, it seems likely that HCUP II indicators will generally remain imperfect measures of quality due to the constraints of the discharge data from which they are constructed. However, it is conceivable that many of these data limitations could be addressed in future versions of HCUP. One weakness involves the non-longitudinal nature of HCUP data; thus, a hospital may lower its inpatient mortality rate by transferring patients to other facilities or other services. Similarly, it is impossible to detect serious complications of treatment that occur after a patient is discharged. These weaknesses could be addressed by encouraging states to provide longitudinally linkable data to HCUP. In fact, many of the states participating in HCUP - including such states as California, Florida, Maryland, New York, and Washington - already collect unique patient identifiers or a combination of variables that can be used to create synthetic identifiers. A second weakness involves the lack of information on important health outcomes, particularly post-discharge mortality. Because of this weakness, it is impossible to track longer-term outcomes that are important for some conditions (e.g., hip fracture), nor is it possible to determine whether differences in discharge policies influence measured mortality rates. These weaknesses could be addressed by encouraging states to provide linked hospital discharge and death index data, or data with personal identifiers that could be linked to the National Death Index.

A third weakness involves the absence of outpatient data; as all but the most intensive therapies continue to move out of the inpatient setting, inpatient data alone provide an increasingly limited view of overall quality of care. With data on ambulatory surgery activity, a much broader range of procedure-related quality indicators would be feasible in HCUP. These would include volume, utilization, and complication or readmission indicators for cardiac catheterization, electrophysiologic stimulation, and a range of arthroscopic and gynecologic procedures. In addition, more complete indicators could be developed for such procedures as angioplasty and laparoscopic cholecystectomy. Nine HCUP states are already collecting such data. With data on emergency department activity, more complete and accurate indicators for quality of care for serious conditions that are sometimes managed on an outpatient basis could be constructed. These conditions include pneumonia, congestive heart failure, angina, and stroke; in the future, acute myocardial infarction might also be included. Emergency department data would also support the construction of more complete quality indicators related to ambulatory care sensitive conditions. One HCUP state is piloting such data collection; expanding this activity to other states is an important goal. In addition to outpatient hospital data, physician office data may provide important information regarding care previous to hospital admissions and other quality. This data could provide a more complete picture of quality for many indicators, particularly the ACSC indicators, where outpatient office care is critical. Nonetheless, to our knowledge the effort to link hospitalizations with outpatient physician office data is far from fruition.

A longer-term but potentially realistic goal is to incorporate more clinical information into HCUP. One approach to doing this would involve data abstracted from electronic medical records. Because such records are neither widespread nor standardized yet, an appropriate focus of research might be to facilitate the adoption of electronic records, and especially to encourage the development of standardized terminology and data structures. It would be helpful to identify a key set of clinical data elements that should be readily abstractable from any electronic medical record system. Other sources of clinical data may be more promising in the shorter run. These include disease registries, such as national registries (e.g. Surveillance, Epidemiology, and End Results (SEER) registries) and state registries of cancer cases. Such data have already been linked to Medicare records, and have proven extremely valuable in providing insights into the quality of care received by cancerpatients. For example, the studies cited in this report on volume-outcome relationships for complex cancer surgeries were based in large part on linked SEER-Medicare data. Another potential source is electronic laboratory information, which for some indicators permits risk adjustment that performs about as well as detailed chart review data. While additional clinical detail in HCUP is certainly a longer-term prospect, it seems clear that electronic clinical data will become a much more central feature of health care delivery in the future. It is not too early to support research to explore the consequences of this trend for HCUP and other electronic data systems that have traditionally been viewed as "administrative."

Many of these recommendations for further HCUP research and development involve enriching HCUP data, or linking it for research purposes to clinically detailed datasets. But such enrichment also leads to increased concerns about data confidentiality and security. Misuse of such data, including the inadvertent or deliberate identification of individual medical histories, is a risk that should not be underestimated. Thus, in contrast to a largely "public use" policy for current HCUP data, such enriched data would require stricter procedures to assure confidentiality. Models for how such data could be handled are provided by HCFA for Medicare data and by some of the HCUP states for the "enriched" versions of their datasets. Briefly, the use of more detailed data requires a progressively more careful review process, to assure that the benefits of the proposed research sufficiently outweigh any associated risks, and to assure that the researchers adhere to strict data securityprotocols (e.g., secure servers, steps to remove confidential information as early in the research process as feasible, IRB evaluation, etc.). HCUP is in a somewhat difficult position on this topic, as its dependence on data produced by states means that data use and security protocols must meet state standards. But AHRQ is in an excellent position to work with states to help them develop appropriate protocols if they have not done so already, and to develop a set of standards for using data from multiple states that meets all state requirements. This is a critical data use issue that goes beyond HCUP: to gain a more complete picture of quality of care, especially for non-Medicare patients, it will be necessary to use data related to quality from multiple states and/or payers. If possible, federal financial support for such activities would be appropriate. The benefits of analyzing data from particular states or payers do not accrue only to their own patients, because such data provide a more complete picture of variation in quality of care that can help to establish benchmarks and guidance for the rest of the country. Moreover, this is a critical time for leadership on the issue of use of confidential data for research purposes. On the one hand, the information involved is very sensitive; on the other hand, to the extent that data from the whole population are not included in studies of health care quality, it is impossible to reach reliable conclusions about opportunities for improving quality of care. The HCUP project provides an opportunity to demonstrate effective ways to balance these concerns, and could be a model for other such collaborative efforts to improve data and research on quality of care and other critical issues affecting the nation's health.