Face validity

Procedure volume is a surrogate measure of quality; its face validity depends on whether a strong association with outcomes of care is both plausible and widely accepted in the professional community.

Abdominal aortic aneurysm repair requires technical proficiency with the use of complex equipment. Technical errors may lead to clinically significant complications, such as arrhythmias, acute myocardial infarction, colonic ischemia, and death. However, we are not aware of any consensus guidelines or recommendations regarding minimum procedure volume.

Precision

Abdominal aortic aneurysmectomy is not as common as the other cardiovascular procedures described in this report; only about 48,600 were performed in the USA in 1997 (1.8 per 10,000 persons). ²⁹³ Based on state all-payer databases, the mean annual frequency of abdominal aortic aneurysmectomies was 16.4-18.3 per hospital in Florida (unruptured only) in 1992-1996 ²⁹⁴ , 8.4 per hospital in New York (unruptured only) in 1990-1995, ²⁹⁵ and 13.8 per hospital in Maryland in 1990-1995. ²⁹⁶

The number of abdominal aortic aneurysm resections is measured accurately with discharge data; in fact, discharge data are probably the best available source for hospital volume information. However, the relatively small number of procedures performed annually at most hospitals suggests that annual volume may be subject to considerable random variation.

Minimum bias

Volume measures are not subject to bias due to disease severity and comorbidities. For this reason, risk-adjustment is not appropriate. Although volume measures are theoretically subject to bias due to variation across hospitals in the use of outpatient surgery facilities, less than 1% of AAA repairs in 1996 were performed in ambulatory settings." ²⁹⁷

Construct validity

Volume is not a direct measure of the quality or outcomes of care. Although higher volumes have been repeatedly associated with better outcomes after abdominal aortic aneurysm resection, these findings may be limited by inadequate risk adjustment.

All but one of 15 studies published since 1985 demonstrated a significant association between either hospital or surgeon volume and mortality after abdominal aortic aneurysm repair. However, three sets of these studies (e.g., Hannan 1989 and 1992^{53, 195}; Kazmers, 1996 and Khuri, 1999^{196, 298}; Pronovost, 1999 and Dardik 1999 and 1998^{62, 197, 296}) appear to include overlapping cases, which may exaggerate the consistency of their results. Two studies of intact aneurysms found that hospital and surgeon volume were significant independent predictors of inpatient mortality, adjusting for other hospital and patient characteristics,^{294, 296} whereas two studies (one of which included ruptured aneurysms) found significant effects only for hospital volume.^{195, 197} The three studies that focused exclusively on ruptured aneurysms reported a significant effect only for surgeon volume.^{62, 195, 299} Six studies that considered only hospital volume found lower mortality at high-volume hospitals;^{188, 196, 295, 300} two of these studies found a significant association among intact aneurysms but not among ruptured aneurysms.^{301, 302}

The only completely negative study was Khuri and colleagues' ²⁹⁸ evaluation of abdominal aortic aneurysmectomies performed in Veterans Affairs hospitals from 1991 through 1997. Their study was the only one that used clinical data, which allowed them to construct a relatively powerful risk-adjustment model. Yet they found only a very weak, nonsignificant association between procedure or specialty volume and risk-adjusted 30-day mortality.

Several studies of this topic have explored whether the volume of something other than the index procedure may be a more powerful predictor of mortality than the volume of the index procedure. The underlying hypothesis is that experience acquired on related, but not identical, cases may lead to improved outcomes. However, two studies failed to show any significant association between the total physician ²⁹⁹ or hospital ⁶² volume of abdominal aortic aneurysmectomies and mortality among ruptured aneurysms, whereas one study showed similar associations between total hospital volume and mortality for both ruptured and intact aneurysms. ¹⁸⁸ Of two studies that evaluated the impact of total vascular surgery volume, one found a significant effect on mortality for both ruptured and intact aneurysms, ³⁰² whereas the other found no effect for intact aneurysms (adjusting for procedure-specific volume). ²⁹⁸ One study found that the hospital volume of surgery for ruptured aneurysms was not associated with postoperative inpatient mortality, but it was associated with fewer inpatient deaths for ruptured aneurysms, suggesting that high-volume hospitals may manage ruptured aneurysms more aggressively. ³⁰³

It is difficult to determine an appropriate hospital volume threshold from the published literature, because numerous studies reported the volume effect only in linear terms^{294, 299} or used volume categories simply to display unadjusted mortality.^{196, 295, 300-302} Among the studies that either reported indirectly standardized mortality rates by volume strata, or used volume categories in multi-level regression models, the recommended hospital volume thresholds were 10, ¹⁹⁵ 20, ¹⁸⁸ or 36 ¹⁹⁷ cases per year of either intact or ruptured aneurysms, or 50 elective repairs of intact aneurysms per 6 years. ²⁹⁶ Among the studies that analyzed volume as a linear effect, but displayed crude mortality by stratum, the stratum definitions were 10 or 21 intact aneurysms per year,^{300, 301} 32 intact aneurysms per 3 years, ¹⁹⁶ or 100 cases of any vascular surgery per year. ³⁰²

Although volume-outcome associations have been demonstrated for abdominal aortic aneurysmectomy, volume seems likely to be both insensitive and nonspecific as a measure of quality. Nonetheless, it has been estimated that shifting patients in California from low-volume to high-volume hospitals would avert 40 deaths per year, given that 64% of all operations are performed in low-volume hospitals. ¹⁹⁰

Fosters true quality improvement

One possible adverse effect of volume-based measures is to encourage low-volume providers (who may also provide poorer quality of care) to increase their volume, simply to reach an artificial threshold. Such responses would probably not improve patient outcomes to the same extent as moving patients from low-volume to high-volume hospitals. For example, Hannan and colleagues ¹⁹⁵ found that the 22 surgeons in New York who increased their volume from 1-6 aneurysm repairs in 1982-84 to 14 or more aneurysm repairs in 1985-87 achieved a minimal decrease in standardized mortality, from 6.8% to 6.2%. The subset of these surgeons who increased their volume to at least 22 cases achieved an important but nonsignificant decrease in standardized mortality, from 5.8% to 2.5%. At the extreme, hospitals may loosen eligibility criteria and perform procedures on patients who are marginal or inappropriate candidates. These arguments would not apply to the subset of ruptured aneurysms, which are not subject to volume manipulation. However, shutting down low-volume hospitals and transferring procedures to high-volume hospitals may worsen outcomes for ruptured aneurysms by delaying surgical intervention.

Prior use

Abdominal aortic aneurysmectomy volume has not been widely used as an indicator of quality. In its Web site, the Pacific Business Group on Health ³⁰⁴ states that "one marker of how well a hospital is likely to perform is the experience of the hospital and its surgical team...in the absence of data to compare hospitals on their complications and survival rates, you can begin evaluating experience by looking at the number of (abdominal aortic aneurysm) surgeries a hospital performs each year." The Center for Medical Consumers posts hospital-specific and operator-specific volumes of "resection of aorta with replacement" for New York hospitals. ³⁰⁵

Procedure Volume

In 1996, 727 hospitals (54.1% of providers) performed at least one procedure. Of these hospitals, the mean (standard deviation) of the number of procedures was 14 (16). The median was 8, and the 90^th and 95^th percentile was 36 and 46, respectively. In general, there are many hospitals with lower volumes and a few hospitals with much higher volumes. Overall procedure volume was stable over the 1995-1997 time period. The mean number of procedures performed in 1995 and 1997 was 14.

Percentage of Procedures at High Volume Hospitals

A moderate to low percentage of procedures were performed at high volume hospitals, depending on which threshold you use. At the threshold 1, 83.9% of AAA repair procedures were performed at 'high volume' providers (and 44.3% of providers are 'high volume'). At the threshold 2, 43.0% were performed at 'high volume' providers (and 12.2% of providers are 'high volume').

Persistence of High Volume

High volume status was moderately persistent over time, depending on the volume threshold used. At threshold 1, 86.2% of high volume providers in 1995 were also high volume in 1996. Similarly, 81.1% of high volume providers in 1996 were also high volume in 1997. At threshold 2, 87.0% of high volume providers in 1995 were also high volume in 1996. Similarly, 75.9% of high volume providers in 1996 were also high volume in 1997.

Construct validity

We estimated the correlation between AAA volume and mortality, adjusting for patient characteristics such as age, sex, and APR-DRG. Volume for carotid endarterectomy is independently and negatively correlated with mortality for carotid endarterectomy (r=-.35, p<.001).

Face validity

Procedure volume is a surrogate measure of quality; its face validity depends on whether a strong association with outcomes of care is both plausible and widely accepted in the professional community.

Carotid endarterectomy (CEA) is a procedure that requires technical proficiency with the use of complex equipment. Technical errors may lead to clinically significant complications, such as abrupt carotid occlusion with or without stroke, myocardial infarction, and death. In two major randomized trials of CEA, researchers pre-selected centers with low rates of perioperative stroke and death, in the belief that these measures reflect surgical skill and quality of care.^{306, 307} As a result, recent professional guidelines focus on the importance of monitoring surgical outcomes, and the avoidance of promoting volume standards. ³⁰⁸

Precision

Publication of the North American Symptomatic Carotid Endarterectomy Trial (NASCET) ²⁴⁷ and the Asymptomatic Carotid Atherosclerosis Study (ACAS) ²⁴⁸ has led to a dramatic increase in the performance of CEA.^249-251 Approximately 144,000 CEAs were performed in the USA in 1997 (5.3 per 10,000 persons). ³⁰⁹ However, many hospitals perform relatively few procedures, suggesting that the actual annual count of procedures may not be a reliable guide to the number of procedures performed on an ongoing basis. For example, 60% of institutions performed fewer than 17 procedures per year in one study of Medicare beneficiaries. ²⁵⁸ Approximately 50% of CEAs performed on Medicare beneficiaries in one state occurred in hospitals performing 21 or fewer operations annually. ⁶⁴ Although these numbers involve Medicare patients only, Medicare beneficiaries comprise approximately 70% of the patients who undergo CEA, so total patient volumes are unlikely to differ substantially.

The number of CEA procedures is measured accurately with discharge data; in fact, discharge data are probably the best available source for hospital volume information.

Minimum bias

Volume measures are not subject to bias due to disease severity and comorbidities. For this reason, risk-adjustment is not appropriate. Although volume measures are theoretically subject to bias due to variation across hospitals in the use of outpatient surgery facilities, less than 1% of CEA surgeries in 1996 were performed in ambulatory settings." ²⁹⁷

Construct validity

Volume is not a direct measure of the quality or outcomes of care. Although higher volumes have been repeatedly associated with better outcomes after CEA, these findings may be limited by inadequate risk adjustment. Only two studies outside the Veterans Affairs system were based on clinical data sets that included "indication for surgery"; both used hospital-specific Medicare volume, not total volume, as the key independent variable.^{64, 310}

Surgeons and hospitals with higher patient volumes tend to have fewer adverse outcomes, including new strokes and deaths.^{64, 258, 310-312} Only two major studies, one limited to Veterans Affairs medical centers ²⁹⁸ and the other from Finland, ³¹³ failed to show a significant hospital volume-outcome relationship for CEA. The magnitude of this effect was impressive in the two studies with the best severity adjustment; for example, Cebul et al. found that undergoing surgery in a high-volume hospital was associated with a 71% reduction in the risk of stroke or death at 30 days, after adjustment for age, gender, indication for surgery, renal insufficiency, and two cardiovascular comorbidities.^{64, 310} In the study by Karp et al., the risk of severe stroke or death was 2.6 times higher at the lowest-volume hospitals than at the highest-volume hospitals.^{64, 310} Other studies with more limited risk adjustment have reported adjusted odds ratios at low-volume hospitals of 1.28 for death ¹⁸⁹ and 2.5-3.1 for "death or definite stroke" (based on ICD-9-CM codes).^{64, 310}

Optimal volume thresholds are difficult to determine, because most studies have only counted Medicare cases. Using a statewide database in New York, Hannan et al. found that hospitals with fewer than 101 cases per year had elevated risk-adjusted mortality. Researchers using Medicare data have reported volume threshold effects at 40^{64, 310} and 62^{64, 310} cases per year, or continuous effects over a volume range from <7 or <11 up to >21 or >50 cases per year.^{64, 310}

Fosters true quality improvement

One possible adverse effect of volume-based measures is to encourage low-volume providers (who may also provide poorer quality of care) to increase their volume, perhaps to reach a threshold of 101 cases per year. Such responses would probably not improve patient outcomes to the same extent as moving patients from low-volume to high-volume hospitals. Indeed, hospitals may loosen eligibility criteria and perform procedures in patients who are marginal or inappropriate candidates. This possibility is worrisome, because the benefits of CEA in asymptomatic patients are modest and easily outweighed by high postoperative complication rates.^{253, 254, 258, 314} Outcomes analyses among Medicare patients undergoing CEA indicate a substantial difference between efficacy and effectiveness.^{64, 250, 258}. Mortality rates among Medicare patients were substantially higher than those reported in the ACAS, even at high-volume centers that participated in the trial, ²⁵⁸ because of more liberal patient selection. For example, the ACAS excluded all patients over 80 years of age, ²⁴⁸ but 15% of the Medicare patients undergoing CEA outside ACAS were in this age range. ²⁵⁸ Patients over 80 years of age experience 2-3 times the perioperative mortality reported for younger patients.^{250, 258-260}

Despite this caveat, previous studies have shown either no relationship between provider volume and patient selection^{258, 298} or a tendency for high volume providers to operate on sicker patients. ⁶⁴ To address this issue, one would ideally consider provider volume in conjunction with the distribution of indications for surgery or major comorbidities. Unfortunately, the indication for surgery is not obtainable from the HCUP database.

The alternative of shutting down low-volume hospitals and transferring procedures to high-volume hospitals may overload these providers and impair access to care.

Prior use

CEA volume has not been widely used as an indicator of quality, although it has been advocated as such. In its Web site, the Pacific Business Group on Health ³⁰⁴ states that "one marker of how well a hospital is likely to perform is the experience of the hospital and its surgical team...in the absence of data to compare hospitals on their complications and survival rates, you can begin evaluating experience by looking at the number of (CEA) surgeries a hospital performs each year." The Center for Medical Consumers posts hospital-specific and operator-specific CEA volumes for New York hospitals. ³⁰⁵

Procedure Volume

In 1996, 904 hospitals (67.2% of providers) performed at least one procedure. Of these hospitals, the mean (standard deviation) of the number of procedures was 52 (60). The median was 31.5, and the 90^th and 95^th percentile was 129 and 169, respectively. In general, there are many hospitals with lower volumes and a few hospitals with much higher volumes. Overall procedure volume was stable over the 1995-1997 time period. The mean number of procedures performed in 1995 and 1997 was 52 and 54, respectively.

Percentage of Procedures at High Volume Hospitals

A moderate percentage of procedures were performed at high volume hospitals. At the threshold 1, 77.8% of carotid endarterectomy procedures were performed at 'high volume' providers (and 37% of providers are 'high volume'). At the threshold 2, 51.0% were performed at 'high volume' providers (and 17% of providers are 'high volume').

Persistence of High Volume

High volume status was highly persistent over time. At threshold 1, 93.5% of high volume providers in 1995 were also high volume in 1996. Similarly, 89.7% of high volume providers in 1996 were also high volume in 1997. At threshold 2, 87.5% of high volume providers in 1995 were also high volume in 1996. Similarly, 87.8% of high volume providers in 1996 were also high volume in 1997.

Construct validity

As we did not retain the carotid endarterectomy mortality indicator due to inadequate precision we were not able to test the construct validity of this indicator. However, CE volume is negatively correlated with several other mortality indicators: CABG (r=-.26, p<.0001), AAA repair (r=-.38, p<.0001) and craniotomy (r=-.18, p<.0001).

Face validity

Procedure volume is a surrogate measure of quality; its face validity depends on whether a strong association with outcomes of care is both plausible and widely accepted in the professional community.

CABG is a procedure that requires technical proficiency with the use of complex equipment. Technical errors may lead to clinically significant complications, such as myocardial infarction, stroke, and death. On the basis of this knowledge and empirical literature (summarized below), the American Heart Association (AHA) and the American College of Cardiology (ACC) have argued for "careful outcome tracking" and supported "monitoring institutions or individuals who annually perform <100 cases." Noting that "some institutions and practitioners maintain excellent outcomes despite relatively low volumes," this panel concluded that "credentialing policies based on conclusions drawn from these data must be made with caution." ¹⁹³ A committee of the Society of Thoracic Surgeons reaffirmed that "until conclusive data become available that link volume to outcome, volume should not be used as a criterion for credentialing of cardiac surgeons...each surgeon should be evaluated on his or her individual results." ³¹⁵

Precision

The frequency of CABG has been relatively stable over the past 15 years, as percutaneous coronary interventions have become more popular. Approximately 366,000 CABG were performed in the USA in 1997 (13.5 per 10,000 persons). ³¹⁶ Several states, including New York, New Jersey, Pennsylvania, and California, have started statewide programs to monitor the frequency and clinical outcomes of CABG surgery. These states differ markedly in mean CABG volume: 627 (range 94-1,814) in New York (1996); 636 (range 111-1,119) in New Jersey (1996-97, annualized); 449 (range 121-1,165) in Pennsylvania (1994-95, annualized); and 266 (range 3-1,643) in California (1996).^{317, 318}

The number of CABG procedures is measured accurately with discharge data; in fact, discharge data are probably the best available source for hospital volume information. The large number of procedures performed annually at most hospitals suggests that annual volume is not subject to considerable random variation (except perhaps in California and similar states). Indeed, Hannan et al. reported year to year hospital volume correlations of 0.96-0.97 in New York. ¹¹⁸

Minimum bias

Volume measures are not subject to bias due to disease severity and comorbidities. For this reason, risk-adjustment is not appropriate. Although volume measures are theoretically subject to bias due to variation across hospitals in the use of outpatient surgery facilities, less than 1% of CABG surgeries in 1996 were performed in ambulatory settings." ²⁹⁷

Construct validity

Volume is not a direct measure of the quality or outcomes of care. Although higher volumes have been repeatedly associated with better outcomes after CABG, these findings may be limited by inadequate risk adjustment. Sowden et al. ³¹⁹ systematically reviewed 15 studies of the volume-outcome relationship for CABG; six used non-overlapping data and reported effect estimates for fixed volume categories. Among these six studies, the apparent benefit of high CABG volume (>200 cases per year) diminished as casemix adjustment improved. Because casemix adjustment was generally more complete in more recent studies, the authors could not exclude the possibility that the benefit of high volume actually decreased between 1972 and 1991.

Using a comprehensive clinical database to adjust for age, gender, unstable angina, ejection fraction, functional class, shock, preoperative intra-aortic balloon pump, recent myocardial infarction, and several comorbidities, Hannan found that the adjusted relative risk of inpatient death at high-volume hospitals (>200 cases per year) in 1989-1992 was 0.84, compared with low-volume hospitals. ¹¹⁸ However, only 3.3% of patients in that study underwent CABG at a low-volume hospital. Another recent study, based on a clinical dataset from the Department of Veterans Affairs, reported a very similar adjusted relative risk of 1.33 at hospitals with less than 101 CABG per year, compared with higher-volume centers. ³²⁰

Older studies using hospital discharge data found larger effects of hospital volume. Differences in risk-adjusted mortality rates across volume quartiles (20-100, 101-200, 201-350, >350 cases per year) were larger for non-scheduled operations (7.7%, 5.5%, 5.9%, and 4.6%, respectively) than for scheduled operations (3.0%, 2.7%, 2.9%, and 2.2%, respectively) in one study. ²²⁸ Analyses using instrumental variables suggested that much, if not all, of the volume effect may be due to "selective referral" of patients to high-quality centers.^{230, 239} Of course, the direction of causation (e.g., higher volume leads to better outcomes, or vice versa) may not affect the validity of using hospital volume as a marker of quality.

Studies of surgeon volume are less directly relevant to the HCUP Quality Indicator project, but one recent study (from New York) demonstrated a statistically significant association between surgeon volume and mortality, which appears to be decreasing over time. ²³³ Specifically, surgeons who performed 50 or fewer CABG in 1989 had a risk-adjusted mortality rate 2.2 times greater than that of surgeons who performed 150 or more CABG. This ratio decreased to 1.89 in 1990, 1.39 in 1991, and 1.36 in 1992. Only in 1989 and 1990 were the mortality differences across surgeon volume categories statistically significant. Two earlier studies of surgeon volume generated counter-intuitive and difficult-to-interpret results.^{227, 321}

Although volume-outcome associations have been demonstrated for CABG, volume seems likely to both insensitive and nonspecific as a measure of quality. For example, Hannan found that some low-volume surgeons achieved outstanding risk-adjusted mortality rates of 2.1% or less in 1992; these surgeons were either transiently low-volume or new to New York State. Nonetheless, it has been estimated that shifting patients in California from low-volume to high-volume hospitals would avert 258 deaths per year. ¹⁹⁰

Fosters true quality improvement

One possible adverse effect of volume-based measures is to encourage low-volume providers (who may also provide poorer quality of care) to increase their volume, simply to reach a threshold of 200 cases per year. Such responses would probably not improve patient outcomes to the same extent as moving patients from low-volume to high-volume hospitals. At the extreme, hospitals may loosen eligibility criteria and perform procedures on patients who are marginal or inappropriate candidates. The alternative of shutting down low-volume hospitals and transferring procedures to high-volume hospitals may overload these providers and impair access to care.

Prior use

CABG volume has not been widely used as an indicator of quality, although specific volume thresholds have been suggested as "standards" for the profession. In its Web site, the Pacific Business Group on Health ³⁰⁴ states that "one marker of how well a hospital is likely to perform is the experience of the hospital and its surgical team...in the absence of data to compare hospitals on their complications and survival rates, you can begin evaluating experience by looking at the number of (CABG) surgeries a hospital performs each year."

Procedure Volume

In 1996, 307 hospitals (26.4% of providers) performed at least one procedure. Of these hospitals, the mean (standard deviation) of the number of procedures was 399 (338). The median was 293, and the 90^th and 95^th percentile was 830 and 1095, respectively. In general, there are a moderate number of hospitals with lower volumes and a few hospitals with much higher volumes. Overall procedure volume grew over the 1995-1997 time period. The mean number of procedures performed in 1995 and 1997 was 375 and 401, respectively.

Percentage of Procedures at High Volume Hospitals

A high percentage of procedures were performed at high volume hospitals. At the threshold 1, 98.3% CABG procedures were performed at 'high volume' providers (and 88% of providers are 'high volume'). At the threshold 2, 90.7% were performed at 'high volume' providers (and 68% of providers are 'high volume').

Persistence of High Volume

High volume status was highly persistent over time. At threshold 1, 99.0% of high volume providers in 1995 were also high volume in 1996. Similarly, 98.1% of high volume providers in 1996 were also high volume in 1997. At threshold 2, 96.9% of high volume providers in 1995 were also high volume in 1996. Similarly, 97.5% of high volume providers in 1996 were also high volume in 1997.

Construct validity

We estimated the correlation between CABG volume and mortality, adjusting for patient characteristics such as age, sex, and APR-DRG. Volume for CABG is independently and negatively correlated with mortality for CABG (r=-.29, p<.001).

Face validity

Procedure volume is a surrogate measure of quality; its face validity depends on whether a strong association with outcomes of care is both plausible and widely accepted in the professional community.

Esophageal cancer surgery requires technical proficiency; errors in surgical technique or management may lead to clinically significant complications, such as sepsis, pneumonia, anastomotic breakdown, and death. However, we are not aware of any consensus guidelines or recommendations regarding minimum procedure volume. The National Cancer Policy Board of the Institute of Medicine and the National Research Council recommends that cancer "patients undergoing procedures that are technically difficult to perform and have been associated with higher mortality in lower-volume settings (including esophagectomy) receive care at facilities with extensive experience (e.g., high-volume facilities)."

Precision

The number of esophagectomies is measured accurately with discharge data; in fact, discharge data are probably the best available source for hospital volume information. Although a few facilities have relatively high volumes, most (e.g., 239 of 273 California hospitals) ¹⁹⁸ perform 10 or fewer esophagectomies for cancer during a 5-year period. As a result, this measure is expected to have poor precision.

Minimum bias

Volume measures are not subject to bias due to disease severity and comorbidities. For this reason, risk-adjustment is not appropriate. Although volume measures are theoretically subject to bias due to variation across hospitals in the use of outpatient surgery facilities, less than 1% of resections in 1996 were performed in ambulatory settings." ²⁹⁷

Construct validity

Volume is not a direct measure of the quality or outcomes of care. Although higher volumes have been repeatedly associated with better outcomes after esophageal surgery, these findings may be limited by inadequate risk adjustment.

Only one study used clinical data to estimate the association between hospital volume and mortality following esophageal cancer surgery. Begg et al. ⁶⁵ analyzed retrospective cohort data from the Surveillance, Epidemiology, and End Results(SEER)-Medicare linked database from 1984 through 1993. The crude 30-day mortality rate was 17.3% at hospitals that performed 1-5 esophagectomies on Medicare patients during the study period, versus 3.9% and 3.4% at hospitals that performed 6-10 and 11 or more esophagectomies, respectively. The association between volume and mortality remained highly significant (p<.001) in a multivariate model, adjusting for the number of comorbidities, cancer stage and volume, and age.

Two other studies using hospital discharge data found similar effects of hospital volume. Using 1990-94 data from California, Patti et al. ¹⁹⁸ estimated risk-adjusted mortality rates of 17%, 19%, 10%, 16%, and 6% across five hospital volume categories (e.g., 1-5, 6-10, 11-20, 21-30, and >30 procedures during the 5-year study period). Their risk adjustment was quite limited; only the year of surgery, age, sex, race, payer source, tumor location, and the total number of secondary diagnoses were included. Using 1990-97 data from Maryland (adjusting only for age and payer source), Gordon et al. ³²² estimated that the adjusted odds of death at minimal-volume (<11 "complex gastrointestinal procedures" per year) and low-volume (11-20 procedures/year) hospitals were 3.8 and 4.0 times that at a high-volume hospital (214 procedures/year). However, the generalizability of these results is limited by the fact that the last category included only one hospital. An older British study found a surgeon volume effect, but did not consider hospital volume. ³²³

Although volume-outcome associations have been demonstrated for esophageal cancer surgery, volume seems likely to both insensitive and nonspecific as a measure of quality. It has been estimated that shifting patients in California from low-volume to high-volume hospitals would avert only 7 deaths per year, although 77% of all operations are performed in low-volume hospitals. ¹⁹⁰

Fosters true quality improvement

One possible adverse effect of volume-based measures is to encourage low-volume providers (who may also provide poorer quality of care) to increase their volume, simply to reach a threshold of 6 cases per year. Such responses would probably not improve patient outcomes to the same extent as moving patients from low-volume to high-volume hospitals. At the extreme, hospitals may loosen eligibility criteria and perform procedures on patients who are marginal or inappropriate candidates. The alternative of shutting down low-volume hospitals and transferring procedures to high-volume hospitals may overload these providers and impair access to care.

Prior use

Esophageal cancer surgical volume has not been widely used as an indicator of quality.

Procedure Volume

In 1996, 265 hospitals (19.7% of providers) performed at least one procedure. Of these hospitals, the mean (standard deviation) of the number of procedures was 2(3). The median was 1, and the 90^th and 95^th percentile was 4 and 6, respectively. In general, there are a moderate number of hospitals with lower volumes and a few hospitals with much higher volumes. Overall procedure volume was stable over the 1995-1997 time period. The mean number of procedures performed in 1995 and 1997 was 2.

Percentage of Procedures at High Volume Hospitals

A low percentage of procedures were performed at high volume hospitals. At the threshold 1, 39.5% esophageal resection procedures were performed at 'high volume' providers (and 8.6% of providers are 'high volume'). At the threshold 2, 34.3% were performed at 'high volume' providers (and 6.4% of providers are 'high volume').

Percentage of Procedures at High Volume Hospitals

A low percentage of procedures were performed at high volume hospitals. At the threshold 1, 39.5% esophageal resection procedures were performed at 'high volume' providers (and 8.6% of providers are 'high volume'). At the threshold 2, 34.3% were performed at 'high volume' providers (and 6.4% of providers are 'high volume').

Persistence of High Volume

High volume status was not persistent over time. At threshold 1, 50.0% of high volume providers in 1995 were also high volume in 1996. Similarly, 58.3% of high volume providers in 1996 were also high volume in 1997. At threshold 2, 57.1% of high volume providers in 1995 were also high volume in 1996. Similarly, 60.0% of high volume providers in 1996 were also high volume in 1997.

Construct validity

We estimated the correlation between esophageal resection volume and mortality, adjusting for patient characteristics such as age, sex, and APR-DRG. Volume for esophageal resection is moderately and negatively correlated with mortality for esophageal resection (r=-.29, p<.05), as well as mortality after other cancer resection procedures.

Face validity

Procedure volume is a surrogate measure of quality; its face validity depends on whether a strong association with outcomes of care is both plausible and widely accepted in the professional community.

Pancreatic cancer surgery requires technical proficiency; errors in surgical technique or management may lead to clinically significant complications, such as sepsis, anastomotic breakdown, and death. However, we are not aware of any consensus guidelines or recommendations regarding minimum procedure volume. The National Cancer Policy Board of the Institute of Medicine and the National Research Council recommends that cancer "patients undergoing procedures that are technically difficult to perform and have been associated with higher mortality in lower-volume settings (including pancreatic resection) receive care at facilities with extensive experience (e.g., high-volume facilities)."

Precision

The number of pancreatectomies is measured accurately with discharge data; in fact, discharge data are probably the best available source for hospital volume information. Although a few facilities have relatively high volumes, most (e.g., 263 of 298 California hospitals) ¹⁹⁹ perform 10 or fewer esophagectomies for cancer during a 5-year period. As a result, this measure is expected to have poor precision.

Minimum bias

Volume measures are not subject to bias due to disease severity and comorbidities. For this reason, risk-adjustment is not appropriate. Although volume measures are theoretically subject to bias due to variation across hospitals in the use of outpatient surgery facilities, less than 1% of pediatric heart surgeries in 1996 were performed in ambulatory settings." ²⁹⁷

Construct validity

Volume is not a direct measure of the quality or outcomes of care. Although higher volumes have been repeatedly associated with better outcomes after esophageal surgery, these findings may be limited by inadequate risk adjustment.

Only one study used clinical data to estimate the association between hospital volume and mortality following esophageal cancer surgery. ⁶⁵ Begg et al. analyzed retrospective cohort data from the Surveillance, Epidemiology, and End Results(SEER)-Medicare linked database from 1984 through 1993. The crude 30-day mortality rate was 12.9% at hospitals that performed 1-5 pancreatic resections on Medicare patients during the study period, versus 7.7% and 5.8% at hospitals that performed 6-10 and 11 or more pancreatic resections, respectively. The association between volume and mortality remained significant (p=.01) in a multivariate model, adjusting for the number of comorbidities, cancer stage and volume, and age.

Eight of the ten studies using hospital discharge data found similar effects of hospital volume. Using 1990-94 data from California, Glasgow and Mulvihill ¹⁹⁹ estimated risk-adjusted mortality rates of 14%, 10%, 9%, 7%, 8%, and 4% across six hospital volume categories (e.g., 1-5, 6-10, 11-20, 21-30, 31-50, and >50 procedures during the 5-year study period). Their risk adjustment was quite limited; only the year of surgery, age, sex, race, payer source, extent of resection, and the total number of secondary diagnoses were included. Using 1990-97 data on radical pancreaticoduoden?ectomies from Maryland (adjusting only for age and payer source), Gordon et al. ( ³²² ) estimated that the adjusted odds of death at minimal-volume (<11 "complex gastrointestinal procedures"/year) and low-volume (11-20 procedures/year) hospitals were 12.5 and 10.4 times that at a high-volume hospital (214 procedures/year). However, the generalizability of these results is limited by the fact that the last category included only one hospital.

Lieberman et al. ³²⁴ used 1984-91 hospital discharge data from New York State to analyze the association between mortality after pancreatic cancer resection and both physician and hospital volumes. Adjusting for the year of surgery, age, sex, race, payer source, transfer status, and the total number of secondary diagnoses, the standardized mortality rate was 19%, 12%, 13%, and 6% at minimal (<10 patients during the 8-year study period), low (10-50 patients), medium (51-80 patients), and high-volume (>80 patients) hospitals, respectively. Surgeon volume was less significantly associated with mortality (6-13% risk-adjusted mortality across 3 volume categories); this effect disappeared in a model that included both physician and hospital volume. The dominance of hospital volume over surgeon volume was confirmed by Sosa et al. ³²⁵ , using Maryland data.

Studies using administrative data from Ontario, ³²⁶ the United Kingdom, ³²⁷ and Medicare ³²⁸ have generated results similar to those from California and New York. The only studies that failed to show a significant hospital volume-outcome association were based on relatively small, nonrepresentative samples from Department of Defense ³²⁹ or major university ³³⁰ hospitals.

Although volume-outcome associations have been demonstrated for pancreatic cancer surgery, volume seems likely to both insensitive and nonspecific as a measure of quality. It has been estimated that shifting patients in California from low-volume to high-volume hospitals would avert only 20 deaths per year, although 57% of all operations are performed in low-volume hospitals. ¹⁹⁰ However, Gordon et al. ³³¹ estimated that 61% of the observed reduction in statewide deaths among patients undergoing the Whipple procedure was attributable to the increasing market percentage of one facility, from 20.7% to 58.5% between 1984 and 1995.

Fosters true quality improvement

One possible adverse effect of volume-based measures is to encourage low-volume providers (who may also provide poorer quality of care) to increase their volume, simply to reach a threshold of 10 cases per year. Such responses would probably not improve patient outcomes to the same extent as moving patients from low-volume to high-volume hospitals. At the extreme, hospitals may loosen eligibility criteria and perform procedures on patients who are marginal or inappropriate candidates. The alternative of shutting down low-volume hospitals and transferring procedures to high-volume hospitals may overload these providers and impair access to care.

Prior use

Pancreatic cancer surgical volume has not been widely used as an indicator of quality.

Procedure Volume

In 1996, 429 hospitals (31.9% of providers) performed at least one procedure. Of these hospitals, the mean (standard deviation) of the number of procedures was 3 (8). The median was 2, and the 90^th and 95^th percentile was 5 and 10, respectively. In general, there are a moderate number of hospitals with lower volumes and a few hospitals with much higher volumes. Overall procedure volume was stable over the 1995-1997 time period. The mean number of procedures performed in 1995 and 1997 was 3.

Percentage of Procedures at High Volume Hospitals

A low percentage of procedures were performed at high volume hospitals. At the threshold 1, 30.3% of pancreatic resection procedures were performed at 'high volume' providers (and 5.1% of providers are 'high volume'). At the threshold 2, 27.0% were performed at 'high volume' providers (and 4.2% of providers are 'high volume').

Persistence of High Volume

High volume status was not persistent over time. At threshold 1, 72.7% of high volume providers in 1995 were also high volume in 1996. Similarly, 71.4% of high volume providers in 1996 were also high volume in 1997. At threshold 2, 73.7% of high volume providers in 1995 were also high volume in 1996. Similarly, 82.4% of high volume providers in 1996 were also high volume in 1997.

Construct validity

We estimated the correlation between pancreatic resection volume and mortality, adjusting for patient characteristics such as age, sex, and APR-DRG. Volume for esophageal resection is moderately and negatively correlated with mortality for esophageal resection (r=-.41, p<.001), as well as mortality after other cancer resection procedures.

Face validity

Procedure volume is a surrogate measure of quality; its face validity depends on whether a strong association with outcomes of care is both plausible and widely accepted in the professional community.

Pediatric cardiac surgery requires technical proficiency with the use of complex equipment. Technical errors may lead to clinically significant complications, such as arrhythmias, congestive heart failure, and death. However, we are not aware of any consensus guidelines or recommendations regarding minimum procedure volume.

Precision

The number of pediatric cardiac procedures is measured accurately with discharge data; in fact, discharge data are probably the best available source for hospital volume information. Previous studies suggest that pediatric cardiac surgery is already highly concentrated at a relatively small number of facilities (e.g., 16 hospitals in New York, 37 in California and Massachusetts together). Although some of these facilities have very high volumes, a significant number (e.g., 16 hospitals in California and Massachusetts) perform fewer than 10 cases per year. The highly skewed volume distribution may have an adverse effect on the precision of this measure.

Minimum bias

Volume measures are not subject to bias due to disease severity and comorbidities. For this reason, risk-adjustment is not appropriate. Less than 1% of pediatric heart surgery are performed on an outpatient basis. ²⁹⁷

Construct validity

Volume is not a direct measure of the quality or outcomes of care. Although higher volumes have been repeatedly associated with better outcomes after pediatric cardiac surgery, these findings may be limited by inadequate risk adjustment.

Only one study used prospectively collected clinical data to estimate the association between hospital volume and mortality following pediatric cardiac surgery. ¹⁹⁴ Hannan et al. ordered all cardiac surgical procedures by their actual mortality rates in the 1992 - 95 Cardiac Surgery Reporting System database. Expert clinicians then grouped the procedures into four clinically sensible subgroups, designed to achieve maximal separation of crude mortality rates (from 1.4% for Category I to 20.1% for Category IV). A multivariate model that included age, complexity category, and four comorbidities (preoperative cyanosis or hypoxia, acidemia, pulmonary hypertension, major extracardiac anomalies) achieved excellent calibration and discrimination (c=0.818). Using this model to estimate risk-adjusted mortality, Hannan et al. found a statistically significant hospital effect (8.26% risk-adjusted mortality at hospitals with fewer than 100 cases per year, versus 5.95% at higher volume hospitals), which was limited to surgeons who performed at least 75 cases per year. Lower volume surgeons experienced relatively high mortality, regardless of total hospital volume. Risk-adjusted mortality differed between low and high-volume hospitals for all 4 complexity categories, although the smallest difference occurred for the highest risk procedures.

Two other studies using hospital discharge data found similar effects of hospital volume. Using aggregated data from California (1988) and Massachusetts (1989), Jenkins et al. ³³² estimated risk-adjusted mortality rates of 8.35% and 5.95% at low-volume (100 or fewer cases) and high-volume (more than 100 cases), respectively. However, they also demonstrated especially high risk-adjusted mortality (18.5%) at very low-volume hospitals with fewer than 10 annual cases, and especially low mortality (3.0%) at very high-volume hospitals with more than 300 annual cases. Jenkins et al. could not evaluate the impact of surgeon volume, but they did report stronger volume effects for higher-risk procedures (e.g., OR=12.1 and 3.2 for Category III-IV procedures at hospitals with <10 and 10-100 annual cases, versus OR=2.4 for Category I-II procedures at hospitals with 10-100 annual cases). Finally, Sollano et al. ²⁹⁵ applied the same 4-category risk adjustment procedure developed by Jenkins to hospital discharge data from New York State in 1990-95. They reported a modest but statistically significant effect (OR=0.944 for each additional 100 annual cases), which was limited to neonates (OR=0.636) and post-neonatal infants (OR=0.720) in stratified analyses.

Although volume-outcome associations have been demonstrated for pediatric cardiac surgery, volume seems likely to both insensitive and nonspecific as a measure of quality. In addition, pediatric cardiac care is already regionalized, so most procedures are performed in medium-to-high volume hospitals. It has been estimated that shifting patients in California from low-volume to high-volume hospitals would avert only 7 deaths per year. ¹⁹⁰

Fosters true quality improvement

One possible adverse effect of volume-based measures is to encourage low-volume providers (who may also provide poorer quality of care) to increase their volume, simply to reach a threshold of 100 cases per year. Such responses would probably not improve patient outcomes to the same extent as moving patients from low-volume to high-volume hospitals. At the extreme, hospitals may loosen eligibility criteria and perform procedures on patients who are marginal or inappropriate candidates. The alternative of shutting down low-volume hospitals and transferring procedures to high-volume hospitals may overload these providers and impair access to care.

Prior use

Pediatric cardiac surgical volume has not been widely used as an indicator of quality.

Procedure Volume

In 1996, 126 hospitals (9.3% of providers) performed at least one procedure. Of these hospitals, the mean (standard deviation) of the number of procedures was 53 (90). The median was 2.5, and the 90^th and 95^th percentile was 149 and 245, respectively. In general, there are many hospitals with lower volumes and a few hospitals with much higher volumes. Overall procedure volume was stable over the 1995-1997 time period. The mean number of procedures performed in 1995 and 1997 was 52.

Percentage of Procedures at High Volume Hospitals

A moderate percentage of procedures were performed at high volume hospitals. At the threshold 1, 75.5% of pediatric heart surgeries were performed at 'high volume' providers (and 21% of providers are 'high volume'). There is no threshold 2 for this procedure.

Persistence of High Volume

High volume status was moderately persistent over time. At threshold 1, 84.6% of high volume providers in 1995 were also high volume in 1996. Similarly, 84.0% of high volume providers in 1996 were also high volume in 1997.

Construct validity

We estimated the correlation between pediatric heart surgery volume and mortality, adjusting for patient characteristics such as age, sex, and APR-DRG. Pediatric heart surgery volume is independently and negatively correlated with mortality (r=−.27, p<.05). However, this analysis does not include the intensive risk adjustment included in the volume studies described in the literature review.

Face validity

Procedure volume is a surrogate measure of quality; its face validity depends on whether a strong association with outcomes of care is both plausible and widely accepted in the professional community.

PTCA is a procedure that requires technical proficiency with the use of complex equipment. Technical errors may lead to clinically significant complications, such as abrupt coronary occlusion with or without myocardial infarction, emergency coronary bypass surgery, and death. On the basis of this knowledge and empirical literature (summarized below), the American Heart Association (AHA) and the American College of Cardiology (ACC) have stated that "a significant number of cases per institution - at least 200 PTCA procedures annually - is essential for the maintenance of quality and safe care." ¹⁹¹ More recent literature (summarized below) led a subsequent expert panel to recommend that "an institution should have an activity level of at least 400 coronary procedures/year...an institution with a volume of <200 procedures/year, unless in a region that is underserved because of geography, should carefully consider whether it should continue to offer the service." ³³³

The same task force expressed concern that "a majority of operators fail to meet the requirements for maintenance of competence, which is a minimum of 75 PTCA procedures performed per year as the primary operator." This standard has been endorsed by the American College of Physicians; ³³⁴ the Society for Cardiac Angiography proposed a lower minimum of 50 cases per year. ³³⁴

Precision

PTCA is an increasingly common procedure; approximately 452,000 were performed in the USA in 1997 (16.7 per 10,000 persons). ³¹⁶ In the Nationwide Inpatient Sample from the Healthcare Cost and Utilization Project, 214 hospitals reported a mean of 382 PTCAs per year in 1993-94. The 27% of hospitals that were classified as low-volume (<200 per year) performed 5% of the procedures, whereas the 31% of hospitals classified as medium-volume (201-400 per year) performed 21% of the procedures and the 42% of hospitals classified as high-volume (>400 per year) performed 74% of the procedures. ¹⁷² Based on state all-payer databases, the mean annual frequency of angioplasties was 226 per hospital in California in 1989 ³³⁵ and 505 per hospital in New York in 1991-1994. ⁶¹

The number of PTCA procedures is measured accurately with discharge data; in fact, discharge data are probably the best available source for hospital volume information. The large number of procedures performed annually at most hospitals suggests that annual volume is not subject to considerable random variation.

Minimum bias

Volume measures are not subject to bias due to disease severity and comorbidities. For this reason, risk-adjustment is not appropriate. Although volume measures are theoretically subject to bias due to variation across hospitals in the use of outpatient surgery facilities, only 7.6% of PTCAs in 1996 were performed in ambulatory settings. ²⁹⁷

Construct validity

Volume is not a direct measure of the quality or outcomes of care. Although higher volumes have been repeatedly associated with better outcomes after PTCA, these findings may be limited by inadequate risk adjustment. Using hospital discharge data to adjust for age, gender, multivessel angioplasty, unstable angina, and 6 comorbidities, high-volume hospitals had significantly lower rates of same-stay coronary bypass surgery (CABG) and inpatient mortality than low-volume hospitals. ¹⁷² Although the magnitudes of the adjusted differences were not reported, the unadjusted differences were modest (e.g., 3.8% versus 4.6% mortality and 4.3% versus 4.6% CABG rates after myocardial infarction, 0.8% versus 1.0% mortality and 2.8% versus 4.0% CABG rates without myocardial infarction). An earlier study using similar data and volume thresholds reported more adverse outcomes (e.g., CABG or death) than expected at low-volume hospitals (e.g., 12.4% observed versus 10.0% expected after myocardial infarction, 6.3% observed versus 5.2% expected without myocardial infarction) and fewer adverse outcomes than expected at high-volume hospitals (e.g., 8.3% observed versus 10.5% expected after myocardial infarction, 4.4% observed versus 5.0% expected without myocardial infarction). ²³⁵ A study based on Medicare data also reported a significant association between hospital volume and mortality, after adjustment for age, sex, race, and year, although no adjusted measures of effect were reported.^{112, 172, 192, 237, 336, 337} Better studies based on clinical data systems (adjusting for left ventricular function) have confirmed higher risk-adjusted mortality and CABG rates at low and medium (<400 cases per year) volume hospitals, relative to high-volume hospitals (e.g., 1.1%versus 0.80-0.95% mortality, 4.2% versus 2.8-3.7% CABG). ⁶¹ A similar study based on clinical data from the Society for Cardiac Angiography and Interventions confirmed the validity of a higher volume threshold than the 200 cases per year originally recommended by the AHA and ACC. Adjusted odds ratios for post-PTCA complications (e.g., death, emergency CABG, or myocardial infarction) were 1.14 at hospitals with 200-399 cases per year, 0.66 at hospitals with 400-599 cases, and 0.54 at hospitals with 600 or more cases. ¹⁹²

Studies of operator volume are less directly relevant to the HCUP Quality Indicator project, but three studies (from New York, Northern New England, and a community hospital in Los Angeles) have supported associations between operator volume and angiographic and clinical success rates, ³³⁶ as well as risk-adjusted same-stay CABG rates.^{61, 64, 258, 310-312} These studies did not demonstrate any association between operator volume and inpatient mortality. Finally, the most recent study showed that the associations between operator volume and both clinical success rates and CABG rates apparently disappeared between 1990-1993 and 1994-1996 in northern New England. ³³⁸

Although volume-outcome associations have been demonstrated for PTCA, volume seems likely to both insensitive and nonspecific as a measure of quality. For example, Hannan demonstrated that low-volume cardiologists can achieve excellent outcomes at high-volume hospitals (e.g., 0.7% risk-adjusted mortality, 2.9% same-stay CABG). ⁶¹ Nonetheless, it has been estimated that shifting patients in California from low-volume to high-volume hospitals would avert 80 deaths per year. This number is consistent with a national estimate of 137 averted deaths and 404 averted same-stay CABG. ¹⁷² It is possible that a low-volume provider may be unavoidable for urgent procedures in less populated areas. But it is unclear whether such urgent cases would do better with (low-volume) PTCA than with alternative, non-PTCA treatments that are not as volume-sensitive.

Fosters true quality improvement

One possible adverse effect of volume-based measures is to encourage low-volume providers (who may also provide poorer quality of care) to increase their volume, simply to reach a threshold of 200 or 400 cases per year. Such responses would probably not improve patient outcomes to the same extent as moving patients from low-volume to high-volume hospitals. At the extreme, hospitals may loosen eligibility criteria and perform procedures in patients who are marginal or inappropriate candidates. The alternative of shutting down low-volume hospitals and transferring procedures to high-volume hospitals may overload these providers and impair access to care.

Prior use

PTCA volume has not been widely used as an indicator of quality, although specific volume thresholds have been suggested as "standards" for the profession. ³³³ In its Web site, the Pacific Business Group on Health ³⁰⁴ (http://www.healthscope.org) states that "one marker of how well a hospital is likely to perform is the experience of the hospital and its surgical team...in the absence of data to compare hospitals on their complications and survival rates, you can begin evaluating experience by looking at the number of (PTCA) surgeries a hospital performs each year." The Center for Medical Consumers posts hospital-specific and operator-specific PTCA volumes for New York hospitals (http://www.medicalconsumers.org).

Procedure Volume

In 1996, 365 hospitals (26.4% of providers) performed at least one procedure. Of these hospitals, the mean (standard deviation) of the number of procedures was 418 (400). The median was 330, and the 90^th and 95^th percentile was 869 and 1157, respectively. In general, there are a moderate number hospitals with lower volumes and a few hospitals with much higher volumes. Overall procedure volume grew over the 1995-1997 time period. The mean number of procedures performed in 1995 and 1997 was 379 and 447, respectively.

Percentage of Procedures at High Volume Hospitals

A moderate to high percentage of procedures were performed at high volume hospitals, depending on the threshold. At the threshold 1, 95.7% of PTCA procedures were performed at 'high volume' providers (and 69% of providers are 'high volume'). At the threshold 2, 77.0% were performed at 'high volume' providers (and 42% of providers are 'high volume').

Persistence of High Volume

High volume status was highly persistent over time. At threshold 1, 97.8% of high volume providers in 1995 were also high volume in 1996. Similarly, 97.9% of high volume providers in 1996 were also high volume in 1997. At threshold 2, 98.5% of high volume providers in 1995 were also high volume in 1996. Similarly, 96.5% of high volume providers in 1996 were also high volume in 1997.

Construct validity

As we did not retain the PTCA mortality indicator due to inadequate precision, we were unable to test the construct validity of this indicator. However, PTCA volume is negatively related to several other post-procedural mortality rates: CABG (r=−.21, p<.001), craniotomy (r=−.200, p<.0001), and AAA repair (r=−.45, p<.0001).

Face validity

The rate of cesarean delivery in the United States increased from 5.5% in 1970 to a high of 24.7% in 1988, with a subsequent decrease to 20.7% in 1996. ³³⁹ Demographic changes in the childbearing population likely account for a relatively small part of this increase, as suggested by Parrish et al. ³⁴⁰ in a study of primary cesarean delivery rates in Washington State from 1987 to 1990. Previous data in population studies have failed to document commensurate improvements in outcomes associated with this increased utilization, ³⁴¹ which has raised questions regarding the appropriateness of current practices. Moreover, cesarean delivery is the most common operative procedure performed in the United States ³⁴² and is associated with higher costs than vaginal delivery. The U.S. Department of Health and Human Services ⁵ and private groups ³⁴³ are encouraging reductions in cesarean rate, the former having set a goal of reducing the cesarean delivery rate to 15% by the year 2000.

While appropriateness of the procedure depends largely on patients' clinical characteristics (see Minimum bias below), studies have shown that individual physician practice patterns account for a significant portion of the variation in cesarean delivery rates.^344-349 Non-clinical factors such as patient insurance status, hospital characteristics, and geographic region have also been related to rates.^350-356

Precision

Burns et al. ³⁴⁷ have shown cesarean delivery is common enough to make good statistical comparisons of hospital and even physician style feasible. Furthermore, the eligible population (pregnant women) is well defined and hospital level reporting reduces the small n problem that may occur with individual providers.

Minimum bias

The overall CS rate cannot determine appropriate use, but the variation in rates across institutions/regions may if the variations do not merely reflect variations in patient disease severity and co-morbidities. Comparison of measures of utilization or outcomes, to be fair, requires adequate adjustment for case mix. ⁸ Studies that have risk-adjusted cesarean delivery rates differ in the risk factors included (indications for cesarean delivery surrounding many risk factors is controversial) and data sources used.

Keller et al. ³⁵⁷ examined singleton births greater than 2500 grams in hospitals in Washington State in 1989 and 1990 using a combination of administrative and birth certificate data. The authors developed separate multivariate models for each of 4 groups (prior cesarean, breech, first birth, other). Risk adjustment including minimal clinical detail (parity, prior cesarean, breech/malposition, placenta/cord problems, active herpes, mother's age, amnionitis, birth weight, sex of child) could explain most of variance among hospitals. The authors concluded that "adjustment of rates did not greatly alter hospital rankings, but the adjustments are fair, improve face validity, and work surprisingly well in explaining which mothers get cesareans. So they should improve the acceptance of monitoring of rates."

In contrast, Aron et al. ³⁵⁸ used data from standardized reviews of medical records to adjust for clinical risk factors in women without prior cesarean section who delivered in the Cleveland area from 1993 to mid 1995. With respect to unadjusted rates, 7 hospitals were statistical outliers. After risk-adjustment, outlier status changed for 5 (24%) of the 21 hospitals. When hospitals were rank-ordered on the basis of cesarean delivery rates, the correlation between unadjusted and adjusted ranking was only moderate. Hospital rankings often changed and, in 12 of the 21 hospitals (57%), the relative difference in unadjusted and adjusted rates was greater than 10%. The authors note that some of the variation in prevalence of risk factors across hospitals may reflect differences in documentation.

Similarly, Bailit et al. ³⁵⁹ showed that risk-adjusting primary cesarean delivery rates using a state birth certificate database substantially changes how hospital performance is judged. Specifically, they found 27% of hospitals with unadjusted rates in the top quartile had adjusted rates that were "risk-appropriate" and that 23% of hospitals with unadjusted rates that were not in the top quartile had adjusted rates that were. In another study using birth certificate data, Glantz ³⁶⁰ also found that crude rates do not accurately reflect the differences in cesarean delivery rates among hospitals and commented "to make judgements regarding clinical practices on the basis of unadjusted rates entails the risk of unwarranted emulation of some hospitals that only appear to have low rates and unfair criticism of some hospitals with seemingly high rates."

Gregory et al ³⁶¹ used discharge data to adjust for clinical risk factors for cesarean delivery in 92,798 singleton Medicaid deliveries in Los Angeles County in 1991. The authors categorized patients according to a revised hierarchical set of indications for cesarean section, ³⁶² revised and validated by ³⁶³ and adjusted for clinical risk factors based on the presence of maternal ICD-9-CM diagnostic codes. The aim of the LA County study was to describe the difference in risk-adjusted cesarean delivery rates according to hospital type; the effect of risk-adjustment on individual hospital rates or on hospital rankings was not evaluated.

Construct validity

We found no studies explicitly evaluating the construct validity of this indicator. In other words, there is no evidence that hospitals with lower cesarean rates more frequently provide better quality of care according to other measures.

As the cesarean rate for "optimal" quality care is unknown, many studies are careful to note that lower cesarean rates do not necessarily reflect better quality care. In some instances, a higher cesarean rate could reflect more appropriate use of the procedure. For example, a meta-analysis by Gifford et al. ³⁶⁴ suggests that elective cesarean delivery for breech presentation may be associated with better neonatal outcomes. Cesarean delivery rates substantially less than the national trend of 90% for infants in a breech presentation may indicate underutilization; however, correlation with maternal and neonatal outcomes would help clarify this issue.

Fosters true quality improvement

The cesarean delivery rate can be decreased by decreasing the primary cesarean delivery rate and/or increasing the vaginal birth after cesarean (VBAC) rate. In some hospitals, one or both of these might result in more maternal and/or infant complications. Sachs et al. ³⁶⁵ note that when a trial of labor after cesarean delivery fails, the rate of maternal morbidity, including infection and operative injuries, increases substantially. The authors cite increasing incidence of the major risk of a trial of labor, uterine rupture, in several states in recent years. However, they caution that, in the absence of chart reviews, one cannot be sure that all these cases involved rupture of a uterine scar from a previous cesarean delivery. Sachs and colleagues also express concern that attempts to decrease the primary cesarean delivery rate may lead to complications associated with higher rates of instrumented (forceps or vacuum-assisted) vaginal delivery. Studies that have compared rates of instrumented vaginal delivery by physicians with low cesarean rates to physicians with higher cesarean rates have found either no significant difference,^{346, 366} or that physicians with low cesarean rates actually use instrumented delivery less. ³⁶⁷ Depending on how rates are risk adjusted, there is the possibility that providers will respond by upcoding diagnoses once measurement begins, though we found no studies where this has been documented. Errors leading to bias are also possible at the data abstraction level.

Prior use

Cesarean section rates were one of the first measures used to judge hospital and health plan performance, ³⁶⁸ and has become one of the most commonly used indicators. Public Citizen's Health Research Group report cesarean delivery rates for 3159 hospitals in 41 states. Cesarean delivery rate is included among the 16 core performance measures in the Maryland Hospital Association's Quality Indicator (QI) Project. ³⁶⁹ Repeat and all cesarean section rates are used by the University Hospital Consortium. ³⁷⁰ Total cesarean delivery rates are used by Florida Agency for Health Care Administration, ³⁷¹ Greater New York Hospital Association, ³⁷² Michigan Hospital Association, ³⁷³ Pacific Business Group on Health, ³⁰⁴ United Health Care, Cleveland Health Quality Choice, ³⁷⁴ JCAHO's IMSystem, Virginia Health Information, ³⁷⁵ Washington State Community Health Information Partnership, ³⁷⁶ HealthGrades.com. ³⁷⁷ In addition, Cesarean section was included in the previous version of HCUP I QIs, and the reduction of cesarean section rate is a goal for Healthy People 2010. ⁵

Precision

This indicator is precise, with a raw provider level mean of 21.4% and a substantial standard deviation of 8.7%. The systematic provider level standard deviation is high, at 4.5%. The provider level variation accounts for a high of total variation, at 1.2%. This means that relative to other indicators, a higher percentage of the variation occurs at the provider level, rather than the discharge level, although more of the variation occurs at the discharge level than for some indicators. Finally, the signal ratio is high, at 88.2%. This means that it is likely that the observed differences in provider performance represent true differences in provider performance, although some of the observed differences are due to unobserved differences in patient characteristics. The R-Square for this indicator is very high at 92.5%, meaning that some additional signal can be extracted using multivariate techniques.

Bias

We did not perform APR-DRG risk adjustment because the categories correspond to the outcome of interest. As a result, the indicator performs well on the multiple measures of minimum bias, using age only (of the mother) risk adjustment. The rank correlation is good at 0.956. Age risk adjustment does seems to impact the lowest and highest decile, with 72.4% of providers remaining after risk adjustment in the highest decile and 79.3% remaining in the lowest decile. There does not seem to be disproportionate impact at either extreme, though the performance in the lowest decile was poorer than that of other indicators. The absolute impact was minimal.

Construct validity

C-section rate loads very highly on factor three, and is inversely related to vaginal delivery after cesarean section and positively related, but to a lesser extent, to incidental appendectomy.

Face validity

The removal of the appendix incidental to other abdominal surgery, such as urological, gynelogical, or gastrointestinal surgeries, is intended to eliminate the risk of future appendicitis, and to simplify any future differential diagnoses of abdominal pain. Controversy about the procedure has abounded since the early 20^th century. ³⁷⁸ Evidence remains unclear for the population as a whole, whether the removal of the appendix increases risk of morbidity and mortality significantly, or whether, given the low risk for future appendicitis, and ease of treatment, it is worth any amount of extra risk. Traditionally, it has been noted that the removal of the appendix may potentially contaminate the other- wise clean operating field.

Unfortunately, the only three published randomized controlled trials did not have sufficient power to exclude a clinically meaningful adverse effect of incidental appendectomy.^379-381 In a prospective, randomized placebo-controlled trial of prophylactic antibiotics, patients who underwent cholecystectomy with incidental appendectomy had a substantially higher wound infection rate than patients who underwent cholecystectomy alone if they did not receive antibiotics (40% versus 16%), but not if they did (9% versus 10%). ³⁸² Two retrospective studies based on large administrative data sets demonstrated significant risk associated with incidental appendectomy among cholecystectomy patients, after adjusting for age, sex, primary diagnosis, and comorbidities. The risk of wound infection was 83% higher among elderly Medicare beneficiaries, ³⁸³ and the risk of any postoperative complication was 53% higher in Ontario general hospitals, ³⁸⁴ when incidental appendectomy was performed. These studies demonstrated substantial selection bias, in that patients selected for incidental appendectomy were younger and had less comorbidity than other cholecystectomy patients. The difference in baseline characteristics was so profound that incidental appendectomy was associated with a 63% decrease in the unadjusted risk of death in Ontario; this effect disappeared after risk-adjustment and reversed after stratification. These results raise serious questions about the validity of other nonrandomized studies of incidental appendectomy, which used smaller samples and inadequate risk adjustment.

Many of the studies discussed above group all patients together regardless of age. However, Andrew and Roty ³⁸⁵ showed that incidental appendectomy was associated with a higher risk of wound infection (5.9% versus 0.9%) among cholecystectomy patients who were at least 50 years of age, but not among younger patients. Based on this finding, and the findings of Warren and colleagues, most commentators believe that incidental appendectomy is inappropriate for elderly patients.^386-388 Although elderly individuals have a higher risk of serious complications due to a perforated appendix, the probability of developing appendicitis is very low, with a lifetime risk of less than 1%. In this age group, it would require at least 115 incidental appendectomies to prevent one hospitalization for appendicitis, and 4,472 incidental appendectomies to avoid a single death.^{383, 389} Given this logic, the risk of incidental appendectomy is believed to outweigh the benefits in the elderly population.

Precision

Rates of incidental appendectomy in the elderly have not been widely studied recently. One 1993 study of Medicare beneficiaries found that about 4% of cholecystectomy cases had a secondary procedure code of incidental appendectomy. ³⁸³ Another study of leading urological surgery departments (non-random sample) found that over 50% of departments did not routinely perform incidental appendectomy during radical cystectomy, regardless of the age of the patient. Fewer than one-third of departments perform it routinely. ³⁹⁰ These findings suggest that incidental appendectomy rates may be difficult to estimate with precision at the majority of hospitals where it is not a routine practice.

Minimum bias

Since incidental appendectomy appears to be contraindicated in an elderly population, very few (if any) cases would be justified by patients' preoperative characteristics. There are documented cases of discovery of diseased appendices during other abdominal surgery, which would justify incidental appendectomy. However, it is unlikely that the number of diseased appendices found incidental to other surgeries would vary systematically across hospitals. There are no identified risk factors that put an individual at higher risk of having an asymptomatic diseased appendix; therefore, it is impossible to estimate the magnitude of bias.

Construct validity

We located no articles explicitly addressing the construct validity of this indicator. Though most of the available evidence appears to contraindicate incidental appendectomy in the elderly, performance of the procedure is subject to patient and surgeon preference. Therefore, incidental appendectomy rates may correlate poorly with other measures of hospital performance. Recent surveys ³⁹⁰ and reviews ³⁸⁷ suggest that incidental appendectomy is still a common practice at some academic centers that enjoy a "high quality" reputation.

Fosters true quality improvement

We found no evidence regarding gaming for this indicator. However, since incidental appendectomy does not generally affect hospital payment, widespread use of this indicator may lead to less frequent coding of the procedure, when it is performed. Since removal of an inflamed appendix is clearly appropriate, it seems unlikely that patients would be denied a necessary appendectomy. Of course, a reduction in the rate of incidental appendectomy may lead to a subsequent increase in the incidence of acute appendicitis.

Prior use

Incidental appendectomy in the elderly is a provider-level utilization indicator in the current HCUP I indicator set. Otherwise, it has not been widely used as an indicator of quality.

Precision

This indicator is precise, with a raw provider level mean of 2.7% and a standard deviation of 3.5%. The systematic provider level standard deviation is moderate, at 1.9%. The provider level variation accounts for a high percentage of total variation, at 1.4%. This means that relative to other indicators, a higher percentage of the variation occurs at the provider level, rather than the discharge level, although more of the variation occurs at the discharge level than for some indicators. Finally, the signal ratio is moderate, at 55.4%. This means that it is likely that some of the observed differences do not represent true differences in provider performance. The moderate R-square for this indicator reflects that relatively modest amount of signal that can be extracted using multi-variate techniques, though these techniques do help somewhat relative to univariate techniques.

Bias

The indicator performs well to very well on the multiple measures of minimum bias. The rank correlation is high at 0.988. Risk adjustment does not appear to impact the extremes of the distribution substantially. Ninety-four percent of providers in the low decile without risk adjustment remain after risk adjustment; 82.9% remain in the highest decile. The absolute magnitude of the impact of risk adjustment is modest.

Construct validity

Incidental appendectomy loads highly on factor 3. It is positively related to cesarean section delivery and negatively related to VBAC.

Face validity

The diagnostic evaluation of patients with presumptive coronary artery disease often involves cardiac catheterization with coronary angiography. Left-sided catheterization provides very useful information about coronary anatomy, as well as left ventricular function and valvular anatomy. Right-sided catheterization is often performed at the same time, but this practice raises two appropriateness issues. First, without a specific indication for right heart catheterization, the clinical yield is extremely low. In the most rigorous prospective study of this phenomenon, case management was changed for only 1.5% of patients who received an incidental right heart catheterization without a listed indication. ³⁹¹ Similar results have been reported from two retrospective studies,^{392, 393} while other studies have failed to distinguish unsuspected right-sided abnormalities that affected management from those that did not. ³⁹⁴ Second, the marginal cost of right heart catheterization has been estimated to exceed $650 per case and $120 million for the nation.

In response to these research findings, the American College of Cardiology and the American Heart Association published guidelines for cardiac catheterization laboratories stating that "without specific indications, routine right heart catheterizations...are unnecessary." ³⁹⁵ Similar guidelines have been published by other medical and public health organizations, such as the Cardiac Advisory Committee of the New York State Department of Health and the Texas Medical Association's Committee on Cardiovascular Diseases.

Precision

In 1996, about 23% of all Medicare beneficiaries who underwent left heart catheterization also underwent right heart catheterization. At the state level, this percentage varied from 11% in Oklahoma to 48% in Massachusetts and 53% in Washington, DC. ³⁹⁶ Given that more than 1.2 million inpatient cardiac catheterizations were performed in the US in 1998, this measure should be estimable with reasonable precision. ³⁹⁷

Minimum bias

Bilateral cardiac catheterization is considered appropriate in the presence of certain clinical indications: suspected pulmonary hypertension or significant right sided valvular abnormalities, congestive heart failure, cardiomyopathies, congenital heart disease, pericardial disease, and cardiac transplantation. The validity of this measure rests on the assumption that the prevalence of these clinical indications is low and/or relatively uniform across the country. Unfortunately, the true prevalence of these indications cannot be reliably derived from administrative data. However, Malone et al ³⁹⁸ found that substantial variation in the use of bilateral catheterization persisted among 37 cardiologists at two large community hospitals, even after adjusting for clinical indications. Bias is likely to account for an even smaller share of variation at the hospital level.

Another source of potential bias is the large number of catheterizations performed on an outpatient basis. In 1996, 472,000 of 1,633,000 catheterizations were performed on an outpatient basis. ²⁹⁷

Construct validity

We located no articles explicitly addressing the construct validity of this indicator.

Fosters true quality improvement

We found no evidence regarding gaming for this indicator. When bilateral cardiac catheterization does not affect hospital payment (as in the DRG system), widespread use of this indicator may lead to less frequent coding of the procedure, when it is performed. It seems unlikely that patients would be denied a bilateral catheterization when the clinical situation clearly warrants it. However, a reduction in the rate of routine bilateral catheterization may lead to rare, but potentially serious, missed diagnoses (e.g., pulmonary hypertension). The long-term significance of missing these rare diagnoses is unclear.

Prior use

Bilateral cardiac catheterization has been widely used as an indicator of quality in the Medicare program. It is one of five quality indicators included in the Medicare Quality of Care Report of Surveillance Measures ³⁹⁹ . From 1993 to 1999, Peer Review Organizations in 20 states developed programs to reduce excessive rates of bilateral cardiac catheterization through education and outreach. Ten of these projects have released results; all documented dramatic utilization changes at the targeted hospitals. It has been estimated that these programs averted at least 6,126 unnecessary bilateral catheterizations. ⁴⁰⁰ Four of these state-based quality improvement projects have been described in the peer-reviewed literature,^401-404 and one documented a spillover effect in the ambulatory setting. ⁴⁰⁵ The results of these studies suggest that right heart catheterization rates represent an actionable opportunity for quality improvement.

Precision

This indicator is very precise, with a raw provider level mean of 19.3% and a substantial standard deviation of 20.0%. The systematic provider level standard deviation is very high, at 16.1%. The provider level variation also accounts for a very high percentage of total variation, at 14.4%. This means that relative to other indicators, a higher percentage of the variation occurs at the provider level, rather than the discharge level. Finally, the signal ratio is very high, at 96.2%. This means that it is likely that the observed differences in provider performance represent true differences in provider performance. The high R-Square demonstrates that a very large proportion of the signal can be extracted using multivariate techniques. However, since the signal ratio is very high to begin with, MSX smoothing adds less additional impact relative to other indicators.

Bias

Signal variance does not change with APR-DRG risk adjustment. The indicator performs well on the multiple measures of minimum bias. The rank correlation is very high at 0.988, and risk adjustment does seem to disproportionately impact the extreme high end relative to the extreme low decile, though the impact is still relatively modest. The absolute impact is also minimal.

Construct validity

Bilateral catheterization rate loads very highly on factor two. It is positively related to CABG mortality and negatively related to laparoscopic cholecystectomy.

Face validity

The rate of cesarean section (CS) in the United States increased from 5.5% in 1970 to a high of 24.7% in 1988; with a subsequent decrease to decreased to 20.7% in 1996. ³³⁹ In the 1980s, the Department of Health and Human Services concluded that encouraging vaginal birth after cesarean section (VBAC) represented a safe way to decrease the overall CS rate ⁴⁰⁶ and subsequently set the following targets for the Healthy People 2000 Objectives: a CS rate < 15 per 100 births, with a primary CS rate of 12 or fewer per 100 births and an increase in the vaginal birth after CS (VBAC) rate to 35 or more per 100 births. This target, as well a number of studies in the literature suggesting that increasing VBAC rates could be safely achieved,^407-412 led to the common adoption VBAC rates as a QI.

Despite the widespread use of VBAC rates as a QI (including in HCUP I), a randomized trial comparing a trial of labor vs. elective repeat cesarean section has yet to appear. Moreover, while physicians and policy makers have presumed that encouraging increased VBAC rates conforms to patient preferences, approximately one third of patients prefer to pursue elective repeat cesarean section.^413-416 In fact, many physicians appear to consider cesarean delivery preferable to vaginal delivery, given the potential complications of the former. ⁴¹⁷ Lastly, a recent article indicates that, accounting for costs and not charges, VBAC is unlikely to achieve significant cost savings compared to repeat CS. ⁴¹⁸

Recommendations for increasing the VBAC rate began to appear in the U.S. in the 1980s. A number of observational studies in the 1980s and early 1990s suggested that a trial of labor in patients with previous cesarean delivery (CD) represented a safe practice.^407-412 More recently, however, evidence for increased risk of maternal and fetal complications associated with this practice has appeared.^{365, 419-424} These complications include uterine rupture, maternal infection and hypoxic injury to the fetus. ³⁶⁵ Uterine rupture represents the most serious of these complications, and a recent study by the CDC indicates that administrative data do not adequately capture this complication. ⁴²⁵

Interestingly, the authors of some studies suggesting an increased risk of uterine rupture associated with a trial of labor have nonetheless concluded that VBAC is overall a safe procedure.^{411, 412, 426, 427} Thus, the policy of recommending VBAC represents to some degree a matter of opinion on the relative risks and benefits of a trial of labor in patients with previous CS. Given the substantial more recent evidence regarding maternal and fetal complications as a result of promoting VBAC and the potential for differences of opinion regarding the significance of these adverse outcomes, the existing HCUP I QI may not be regarded as a straightforward corrective for the previous (presumed) overuse of CS. Given the concern regarding maternal and fetal safety, some have advocated for further research to establish the "right rate" of VBAC. ⁴²⁸

Precision

We located no evidence on the precision of this indicator. However, VBAC is a relatively common procedure, and thus we would expect it to be measured with good precision.

Minimum bias

In one study, ⁴²⁹ only 42.0% of women with a CS-Vaginal delivery sequence were correctly identified on the second birth certificate as a VBAC. Only 75% of 25,491 women from 1980 through 1988 with a previous cesarean were so designated on the birth certificate; 80% of women with a V- CS sequence were correctly designated as primary cesarean. Although this study employed birth certificates, these findings suggest that administrative data accurately distinguish the mode of current mode of delivery (vaginal vs. CS), but less accurately identify VBAC and primary cesarean delivery.

The proportion of patients with certain sociodemographic profiles ⁴³⁰ and medical indications for CS or contraindications to a trial of labor^{358, 360, 431-434} exerts a significant impact on QIs related to rates of CS and VBAC. Although adjusting for case-mix does not eliminate wide variation in rates of CS ⁴³⁵ , unadjusted rates only modestly correlate with adjusted rates ³⁵⁸ . Administrative data sources, such as vital records and hospital discharge data, do not include the clinical factors required to identify appropriate candidates for trial of labor.^{358, 432, 436, 437} Thus, the denominator for VBAC rates calculated using administrative data will include women with an accepted medical indication for repeat CS delivery (e.g., women with a prior classic CS), not to mention patients who make an informed decision not to pursue a trial of labor. ⁴¹⁶

Prompted by concern over an increase uterine rupture (UR) the Massachusetts Department of Public Health and the CDC conducted an investigation of the validity and reliability of ICD-9-CM codes in hospital discharge data to identify UR cases. The study covered maternal discharges from Massachusetts hospitals from 1990 through 1997; women with and without a history of prior CS were included. Potential cases of UR were identified with an ICD-9-CM diagnostic code in any of the 10 diagnostic fields of 665.0 ("rupture of uterus before onset of labor"), 665.1 ("rupture of uterus during labor," including "rupture of uterus not otherwise specified"), or 674.1 ("disruption of cesarean wound," including "dehiscence or disruption of uterine wound"). Two clinicians then reviewed the medical records of suspected cases to confirm the diagnosis of UR (defined as any unintentional disruption of the uterine wall in a pregnant woman regardless of cause, size, degree of severity, or location).

The design of the study does not permit estimation of the sensitivity of the codes for UR. Positive predictive values (PPVs) were calculated as the number of confirmed cases divided by the number of reviewed suspected cases multiplied by 100. The average PPV during the 8-year period was 50.7% for ICD-9-CM codes 665.0 and 665.1 and 28.6% for code 674.1. The overall PPV of the three codes was 39.8%. Approximately half of the uterine ruptures that result from a trial of labor (among patients with a prior CS) are coded to 674.1 rather than 665.0/1. Furthermore, only about half of the patients coded as having UR (665.0/1) actually had this complication. As an editorial note accompanying the study point out, these coding problems may reflect the fact that the development of the ICD-9 system predates the interest in monitoring trends in the incidence of UR associated with increased VBAC rates. As ICD-10 codes were also published prior to increased concern over UR, administrativedata will have limited use in monitoring the potential negative impacts of attempted VBAC.

Construct validity

The likelihood that a patient will undergo VBAC correlates with certain provider and institutional variables,^{354, 361, 433, 438, 439} suggesting that certain providers are more likely to adapt to changes in policy or technology.

Fosters true quality improvement

Promotion of VBAC a QI has led to successful increases in the VBAC rate in some cases,^{440, 441} but not in others. ⁴⁴² The major opportunity for gaming (i.e., spurious improvement) lies in focusing on patients more likely to successfully complete a trial of labor after previous CS.

Prior use

VBAC is one of the most commonly used indicators. In addition to being included in the previous version of the HCUP I indicator set, VBAC is included in and used by JCAHO's IM System, Maryland QI Project, ³⁶⁹ Michigan Hospital Association, ³⁷³ HealthGrades.com, ³⁷⁷ Cleveland Health Quality Choice, ³⁷⁴ and the University Hospital Consortium. ³⁷⁰ Further, JCAHO has selected VBAC as one of its core measures. ⁴⁴³

Precision

This indicator is very precise, with a raw provider level mean of 33.6% and a substantial standard deviation of 14.8%. The systematic provider level standard deviation is very high, at 11.7%. The provider level variation accounts for a high percentage of total variation, at 5.7%. This means that relative to other indicators, a higher percentage of the variation occurs at the provider level, rather than the discharge level, though some variation remains at the discharge level. Finally, the signal ratio is high, at 83.1%. This means that it is likely that the observed differences in provider performance represent true differences in provider performance, although some of the observed difference is due to unobserved differences in patient characteristics. The high R-Square indicates that some additional signal can be extracted using multivariate techniques, though this additional impact is moderate.

Bias

Signal variance does not change with risk adjustment. We did not perform APR-DRG risk adjustment because the categories correspond to the outcome of interest. As a result, the indicator performs very well on the multiple measures of minimum bias, adjusting only for the age of the mother. The rank correlation is very high at 0.995. Age risk adjustment does not seem to impact disproportionately at the extreme high and low end, though some there is some modest impact at the low extreme relative to other indicators. The absolute change is low, and no providers change more than two deciles with risk adjustment.

Construct validity

VBAC rate loads very highly on factor three, and is inversely related to cesarean section delivery and to a lesser extent incidental appendectomy.

Face validity

Cholecystectomy, surgical removal of the gallbladder, is now generally performed with a laparoscope (in about 75% of uncomplicated cases in the NIS). ⁴⁴⁴ One of the largest randomized controlled trials (RCTs) comparing cholecystectomy by laparoscopy versus minilaparotomy found that the laparoscopic procedure was associated with less postoperative pain, lower patient-controlled morphine consumption, better postoperative pulmonary function and oxygen saturation, and quicker return to leisure activities, work in the home, and social activities (but no difference in return to employment).^445-451 This study also confirmed that the laparoscopic approach is associated with less postoperative narcotic use, less sick leave, and shorter convalescence (even with conversion rates as high as 13%). Observational studies suggest that laparoscopic surgery also has a lower mortality rate^452-456 and a lower readmission rate.^452-454 Only one large RCT reported no difference in hospital stay, time off work, or return to full activities. ⁴⁵⁷ The authors attribute this difference to their small-incision (7 cm) minilaparotomy and their unusual blinding method, in which patients were told to start eating, get out of bed, and go home whenever they felt ready, without knowing which procedure they had received.

As a result of these studies, laparoscopic cholecystectomy is now widely accepted as the "standard of care for patients requiring cholecystectomy," ⁴⁵⁸ in the absence of specific contraindications (e.g., coagulopathy, late pregnancy, morbid obesity, cirrhosis). However, this procedure requires more technical skill than the open approach. Thus, a higher rate of laparoscopic cholecystectomy (as a proportion of all cholecystectomies) suggests that a hospital can rapidly achieve proficiency in up-to-date treatment methods.

Precision

Cholecystectomies are relatively common, so moderately precise estimates of differences in laparoscopic utilization across hospitals can be obtained. In the NIS, the average number of cholecystectomies per hospital is approximately 70, with nearly two-thirds of all hospitals performing at least 3 procedures. Still, random variation in laparoscopic utilization across hospitals in a particular year may be considerable, particularly for hospitals that perform few procedures. Restricting the denominator definition to uncomplicated cases reduces the size of the denominator by about 20%, which further reduces precision. ⁴⁵⁹

Minimum bias

The current HCUP I definition limits the denominator to uncomplicated cases: non-acute cholecystitis (inflammation of the gallbladder) and/or cholelithiasis (formation of bile stones in the gallbladder). For these patients, cholecystectomy is an elective procedure. Cholecystectomies on patients with acute cholecystitis are performed as emergency procedures, which increases the risk of iatrogenic injury. ⁴⁶⁰ Higher risks of complications with LC are also associated with older age and the presence of common bile duct stones. ⁴⁵⁹ Only a limited number of surgeons are comfortable with laparoscopic common bile duct exploration, which is described by the Society of American Gastrointestinal Endoscopic Surgeons as "a complex biliary procedure that demands a well-trained operating room team and facilities and equipment beyond that required for routine LC." In addition, as surgeons become more experienced in LC, they are more likely to perform LC on more difficult patients, such as those with acute cholecystitis ⁴⁶¹ and those at older and younger ages.^{462, 463} These examples illustrate that patient referral patterns and other selection factors may lead to substantial differences in laparoscopy rates (as a proportion of all cholecystectomies) across hospitals. While many of these patient characteristics can be measured and thus adjusted out in comparisons across hospitals, controlling for other aspects of patient severity may be more difficult.

In 1996, there were an estimated 770,000 cholecystectomies, both open and laparoscopic, in the US - 322,000 ambulatory (42%) and 448,000 inpatients (58%). ²⁹⁷ Thus, use of only inpatient data could be substantially biasing, in that it eliminates those cholecystectomies performed on an outpatient basis, most of which are likely to be laparoscopic.

Construct validity

We found no studies explicitly evaluating the construct validity of this indicator. In other words, there is no evidence that hospitals that use the laparoscopic approach more frequently provide better quality of care, according to other measures. One study examined factors associated with the rate of hospital adoption of laparoscopic cholecystectomy in Pennsylvania; the authors reported that "nearly universal adoption occurred at a speed not previously reported" (e.g., within 2 years). Participation in residency training was the only independent predictor of earlier LC adoption.

Fosters true quality improvement

This indicator raises several concerns related to perverse incentives. Within 3 years after its introduction, 76% of cholecystectomies in Maryland^{462, 463} and 72% of those in Pennsylvania ⁴⁶⁴ were performed laparoscopically. In both states, and in New York, ⁴⁶⁵ the advent of laparoscopic surgery led to a substantial (28-34%) increase in the overall cholecystectomy rate, especially involving uncomplicated and elective patients.^466-468 Similar, but less dramatic, increases have been reported from the Veterans Affairs system (10%), ⁴⁵⁶ Canada (17%), Australia (26%), ⁴⁶⁹ and Scotland (19%). ⁴⁷⁰ These trends eliminated most of the expected decline in cholecystectomy-associated deaths in Maryland and in the Veterans Affairs system. The NIH Consensus Development Panel tried to slow this phenomenon by declaring that "the availability of laparoscopic cholecystectomy should not expand the indications for gallbladder removal." ⁴⁶⁶ It is not clear whether this declaration has had any moderating effect on community practice. However, providers could readily boost LC rates by recruiting more persons with marginal clinical indications to undergo cholecystectomy.

The other concern in this domain is that the "optimal" LC rate has not been defined. Provider experience may be an important, and desirable, determinant of how the procedure is used across a wide range of clinical situations. At some hospitals, increasing proportionate LC utilization above 75% might lead to more biliary tract or intestinal complications. In addition, previous studies have clearly demonstrated a learning curve, whereby surgeons' outcomes and operating times improve as they gain experience with LC.^{471, 472} Technical complication rates appear to stabilize at a low rate after about 75 procedures. ⁴⁷³ Incentives to increase LC utilization may have negative consequences if local physicians lack appropriate training and expertise.

These findings suggest that proportionate LC utilization must be interpreted together with overall cholecystectomy rates (to rule out over-referral of patients who are inappropriate or marginally appropriate) and LC complication or conversion rates (to rule out inappropriate use of LC among patients who would benefit from open surgery).

Prior use

The rate of laparoscopic cholecystectomy is a current indicator in the HCUP I QI set. We were unable to find evidence that this measure has been used as a quality indicator in other settings. Indeed, the rapid and nearly universal adoption of LC that occurred between 1990 and 1993 makes future implementation of this measure seem unlikely.

Precision

This indicator is very precise, with a raw provider level mean of 66.2% and a substantial standard deviation of 19.2%. The systematic provider level standard deviation is very high, at 13.3%. The provider level variation also accounts for a very high percentage of total variation, at 7.9%. This means that relative to other indicators, a higher percentage of the variation occurs at the provider level, rather than the discharge level. Finally, the signal ratio is high, at 89.1%. This means that it is likely that the observed differences in provider performance represent true differences in provider performance, although some of the observed differences are due to unobserved differences in patient characteristics. The high R-square demonstrates that a large proportion of signal can be extracted using multivariate techniques, though not as much as some other indicators. The additional impact of multivariate techniques is modest.

Bias

Signal variance does not change with risk adjustment. We did not perform APR-DRG risk adjustment because the categories correspond to the outcome of interest. As a result, the indicator performs very well on the multiple measures of minimum bias, using age-sex adjustment. The rank correlation is high at 0.966. Age and sex risk adjustment does seem to disproportionately impact the low extreme relative to the high extreme of the distribution; ninety-four percent of providers in the high decile remain after risk adjustment, while only 78.8 of providers in the low decile remain. The absolute change is low, and relatively few providers change more than two deciles with risk adjustment.

Construct validity

Laparoscopic cholecystectomy loads very highly on factor two. It is inversely related to bilateral catheterization, and CABG mortality rate.

Face validity

Coronary artery bypass graft (CABG) surgery is performed on patients with coronary artery disease (CAD). Most previous studies of small area variation have found relatively high variation in CABG rates, as noted by the systematic component of variation (.758). ⁴⁷⁴ This systematic component of variation (SCV) "compares geographic variability between DRGs after removing random effects. ⁴⁷⁴ This variation is not been explained by population characteristics such as age and race.

The clinical indications for CABG in patients with symptoms less major than three-vessel disease, previous myocardial infarction, or less than strongly positive exercise ECG tests are unclear.^{262, 475} No randomized controlled trials have demonstrated that CABG improves clinical outcomes in patients with some combination of these indications.

Precision

Because adult admissions for CABG are relatively common, it should be possible to generate precise estimates of utilization at the area level. However, random variation in utilization rates may become more problematic for relatively small areas (e.g., zip codes) or underpopulated areas (e.g., rural counties).

Minimum bias

Utilization rates standardized at the area level (e.g., adult population of the county or SMSA) may be biased by differences in the prevalence of CAD. The prevalence of CAD may, in turn, be related to the age structure of the population and the prevalence of behavioral or physiologic risk factors such as smoking and hyperlipidemia. Even though race and demographic factors have significant effects on the likelihood of CABG, previous studies have shown that sociodemographic differences account for very little of the observed variation in CABG rates. ²⁶² While one study reports no significant difference between age categories and appropriateness, ²⁶² another reports that "patients 75 years of age and older were more likely to have surgery classified as uncertain than were younger patients." ⁴⁷⁵ Although there is some report of variation based on age, it is not likely that this factorexplains all the variation observed in CABG rates. Some differences in CABG rates across areas may be attributable to the referral of rural and other patients from outside the area for surgery; however, such referrals are unlikely to explain a large part of the substantial differences in rates across small geographic areas.

Construct validity

For this indicator to perform well in identifying true quality of care problems, there must be evidence of significant inappropriate or questionable utilization in population-based studies. In addition, there must be substantial variation in the rate of inappropriate utilization across providers or small areas. Previous studies have classified possible indications for CABG as appropriate, uncertain, or inappropriate, as a method for assessing quality of care.

In a follow-up of a NY appropriateness study, a panel of cardiologists from Duke University reviewed 308 cases. In their results, the rate of inappropriate procedure increased to 6% and the rate of uncertain procedures increased to 12%. ⁴⁷⁶ In another study of 12 Academic Medical Center Consortium hospitals, the rate of CABG for inappropriate indications was as much as 1.9% overall, ranging from 0 to 5% across hospitals (P=0.02). The rate of CABG for uncertain indications was 7%, ranging from 5 to 8% across hospitals. ⁴⁷⁵ In 15 randomly selected non-federal hospitals in New York State (which has relatively low CABG rates by US standards), 2.4% of CABG was inappropriate and 7% uncertain. These rates also varied among hospitals. ²⁶²

In 319 randomly selected patients with CABG between July 1987 and June 1988, 16% overall received CABG for inappropriate reasons and 26% received the procedure for uncertain reasons. ³¹⁷

In a comparison of Canadian and New York State samples, appropriateness of CABG was measured according to criteria from each nation. Based on both Canadian and US criteria, the Canadian sample held more cases of CABG for uncertain indications. However, based on Canadian criteria, the Canadian sample had fewer cases of CABG performed for inappropriate indications but, based on US criteria, the rate of inappropriate CABG in both samples is lower than that Canadian criteria assess for either of them, and the rates are approximately the same. ⁴⁷⁷

In a study of seven of eight public Swedish heart centers, which perform 92% of all bypass surgeries in Sweden, it was found that 9.7% of all CABG procedures were performed for inappropriate indications and 12.3% of procedures were performed for uncertain indications. ⁴⁷⁸ Finally, among 153 catheterization patients referred to either a university cardiac laboratory or a VA cardiac lab in Maryland, the rate of CABG performed for inappropriate indications ranged from 17 to 46%, based on three different appropriateness criteria: RAND, ACC/AHA, or RAS. The rate of CABG for uncertain indications was 17%, based on RAND criteria, the only set of criteria that accounts for this rating.

Thus, though most studies have found relatively low rates of inappropriate use, there is some evidence of variation in inappropriate rates across geographic areas. Moreover, a larger proportion of bypass surgery procedures are performed for indications in which benefits are uncertain; procedure rates for uncertain indications may also vary substantially across hospitals and areas.

Fosters true quality improvement

Little evidence exists on whether the use of CABG as a quality indicator might differentially reduce procedures which are inappropriate or of unclear benefit, rather than appropriate procedures.

Prior use

The hospital-based rate of CABG is a current indicator in the HCUP I QI set. The area-based rate of CABG is a current indicator in the Dartmouth Atlas. ⁴⁷⁹

Precision

This indicator is moderately precise, with a raw area level mean is 180.4 per 100,000 population and a standard deviation of 571.6. The systematic area level standard deviation is very high, at 0.25%. The area level variation accounts for a very high percentage of total variation, at 0.21%. This means that relative to other indicators, a larger percentage of the variation occurs at the area level, rather than the discharge level. The signal ratio is very high, at 97.3%. This means that it is very likely that the observed differences in area performance represent true differences in area performance. The R-Square is also very high, demonstrating that a large proportion of signal can be extracted with either univariate or multivariate techniques.

Bias

Signal variance does not change substantially with risk adjustment. The indicator performed relatively poorly on most measures of minimum bias, using age and sex adjustment. Both the relative and absolute impact of risk adjustment is substantial. The exception is that risk adjustment did not effect the lowest decile substantially, with over 95% of providers in the lowest decile remaining after risk adjustment.

Construct validity

CABG rate loads on factor 1, and is related to all other area utilization indicators.

Face validity

Hysterectomy is performed on patients with a number of indications, such as recurrent uterine bleeding, chronic pelvic pain, or menopause, usually in some combination. Small area variation has been noted. One study of variation within the state of Maryland found relatively moderate variation, as noted by the systematic component of variation (.083). ⁴⁷⁴ This systematic component of variation (SCV) "compares geographic variability between DRGs (diagnosis-related groups) after removing random effects." ⁴⁷⁴

The clinical indications for hysterectomy that include persistent or recurrent abnormal bleeding, pain, an adnexal mass, limited hormonal therapy, and premenopausal age are unclear. No randomized controlled trials have demonstrated that hysterectomy improves clinical outcomes in patients with uncertain clinical indications.

Precision

Because adult admissions for hysterectomy are relatively common, it should be possible to generate precise estimates of utilization at the area level. However, random variation in utilization rates may become more problematic for relatively small areas (e.g., zip codes) or underpopulated areas (e.g., rural counties).

Minimum bias

Utilization rates standardized at the hospital level (e.g., all adult discharges) were included in earlier versions of the HCUP I QIs, but are likely to be biased by local market characteristics and referral patterns. In other words, hospitals that specialize in specific services, such as elective or gynecological surgeries, may appear to have higher utilization rates than hospitals that refer such patients elsewhere. Theoretically, utilization rates standardized at the area level (e.g., adult population of the county or SMSA) may be biased by differences in the prevalence of those indications that warrant the procedure. The prevalence of these indications may, in turn, be related to the age structure of the population and the prevalence of behavioral or physiologic risk factors. Previous studies have shown that observed variation in hysterectomy rates may be accounted for by sociodemographic differences in the patient population. In a study of seven managed care organizations, "older women were more likely than younger women to have received a hysterectomy for appropriate reasons." ²⁶³ In a study of women in the UK, "the risk of hysterectomy was significantly related to parity." ⁴⁸⁰ "Only 5% percent of nulliparous women had a hysterectomy; the risk for women with one to four children varied between 8% and 11%, and the risk for those with five or more children was 31% (P=0.002)." ⁴⁸⁰ Even after adjusting for parity in a regression model, "the risk of hysterectomy was still significantly associated with educational qualifications." ⁴⁸⁰

Construct validity

For this indicator to perform well in identifying true quality of care problems, there must be evidence of significant inappropriate utilization in population-based studies. In addition, there must be substantial variation in the rate of inappropriate utilization across providers or small areas. Previous studies have classified possible indications for hysterectomy as appropriate, uncertain, or inappropriate, as a method for assessing quality of care.

In a random sample of 642 hysterectomies (non-emergency and non-oncological), 16% of procedures were deemed inappropriate, based on patient indications, while 25% were deemed uncertain. The rate of inappropriate indications for hysterectomy varied across plans from 10% to 27%. ²⁶³

Another study focused on women receiving hysterectomies in Southern California. The rate of overall inappropriate indications was 70%, varying from 45% to 100% across diagnoses indicative of hysterectomy. Uncertain indications were not evaluated. ⁴⁸¹

Fosters true quality improvement

In theory, use of this quality indicator might reduce appropriate as well as inappropriate hysterectomies. Little evidence exists on whether this is likely to occur, or on the extent to which overall hysterectomy rates are correlated with inappropriate hysterectomy rates.

Prior use

The hospital-based rate of hysterectomy is a current indicator in the HCUP I QI set. The area-based rate of hysterectomy is a current indicator in the Dartmouth Atlas. ⁴⁷⁹

Precision

This indicator is precise, with a raw area level rate of 419.4 per 100,000 population and a substantial standard deviation of 323.3. The systematic area level standard deviation is very high, at 0.19%. The area level variation accounts for a high percentage of total variation, at 0.10%. This means that relative to other area indicators, a higher percentage of the variation occurs at the area level, rather than the discharge level, though some remains at the discharge level. The signal ratio is very high, at 93.6%. This means that it is likely that the observed differences in area performance represent true differences in provider performance. The very high R-square represents the high proportion of signal that can be extracted using multivariate techniques. However, multivariate techniques have little additional impact on the amount of variance that can be extracted from this indicator, since the signal ratio is already very high.

Bias

Signal variance does not change substantially with risk adjustment. The indicator performed well on most measures of minimum bias, though the only adjustment was for age. Age adjustment does not impact the absolute ranking substantially. No providers change more than 2 deciles in performance. Risk adjustment appears not to impact the relative rankings of areas substantially.

Construct validity

Hysterectomy loads on factor 1, and is related to all other area utilization indicators.

Face validity

Laminectomy is performed on patients with a herniated disc or spinal stenosis. Most previous studies of small area variation have found relatively high variation in laminectomy rates, as noted by the systematic component of variation (.292). ⁴⁷⁴ This systematic component of variation (SCV) "compares geographic variability between DRGs [diagnosis-related groups] after removing random effects. ⁴⁷⁴ Another study in Maryland found a 0.50 variation rate [(sample standard deviation)/(sample mean)] in laminectomy for selected Maryland service areas. ⁴⁸² A study of Boston-area hospitals found a 2.2-fold variation in laminectomy rates among districts. ⁴⁸³ Larequi-Lauber et al. report that, in the United States, "the use of back surgery varies from one area to another by as mush as 15-fold." ⁴⁸⁴ This high variation has not been explained by population characteristics such as age and socioeconomic status.

The clinical indications for laminectomy in patients with minor neurological findings, lengthy restricted activity, and equivocal imaging for discal hernia or spinal stenosis are unclear. No randomized controlled trials have demonstrated that laminectomy improves clinical outcomes in patients with these uncertain indications.

Precision

Because adult admissions for laminectomy are relatively common, it should be possible to generate precise estimates of utilization at the area level. Indeed, even after accounting for random variations, large differences in area laminectomy rates remain. ⁴⁷⁴ However, random variation in utilization rates may become more problematic for relatively small areas (e.g., zip codes) or sparsely populated areas (e.g., rural counties).

Minimum bias

Utilization rates standardized at the area level (e.g., adult population of the county or SMSA) may be biased by differences in the prevalence of herniated disc or spinal stenosis. The prevalence of herniated disc or spinal stenosis may, in turn, be related to the age structure of the population and the prevalence of behavioral or physiologic risk factors. However, previous studies have shown that sociodemographic differences and other measurable population characteristics account for very little or none of the observed variation in laminectomy rates. ⁴⁸²

Construct validity

For this indicator to perform well in identifying true quality of care problems, there must be evidence of significant inappropriate utilization in population-based studies. In addition, there must be substantial variation in the rate of inappropriate utilization across providers or small areas. Previous studies have classified possible indications for laminectomy as appropriate, uncertain, or inappropriate, as a method for assessing quality of care.

In an assessment of cases at one Swiss hospital, 23% of patients received surgical treatment for herniated discs for inappropriate reasons and 29% received surgical treatment for uncertain indications. ⁴⁸⁵ In another study of teaching hospital patients undergoing surgery for herniated disc or spinal stenosis (lumbar discectomy or spinal stenosis surgery), 38% of surgeries were performed for inappropriate indications. Uncertain indications were combined with "appropriate" category.

Fosters true quality improvement

Little evidence exists on whether use of area laminectomy rates as a quality indicator would lead to less performance of laminectomies for inappropriate or uncertain indications, without reducing the use of laminectomy for appropriate indications.

Prior use

The hospital-based rate of laminectomy is a current indicator in the HCUP I QI set. The area-based rate of laminectomy is a current indicator in the Dartmouth Atlas. ⁴⁷⁹

Precision

This indicator is moderately precise, with a raw area level mean of 139.0 per 100,000 population and a standard deviation of 347.5. The systematic area level standard deviation is high, at 0.13%. The area level variation accounts for a very high percentage of total variation, at 0.10%. The signal ratio is very high, at 96.7%. This means that it is very likely that the observed differences in area performance represent true differences in area performance. Multivariate smoothing does not have additional impact in extracting signal, mainly due to the already very high ratio.

Bias

The signal variance does not change with age-sex risk adjustment. The indicator performed fairly to well on most measures of minimum bias. Risk adjustment does not impact the absolute performance of areas substantially (6.3%). Only 7.4% of areas changed more than 2 deciles. Risk adjustment appears to impact the highest decile disproportionately to the lowest decile, with 31.8 percent of areas in the highest decile remaining after risk adjustment, compared to 95.5% in the lowest decile. However, the overall relative impact of risk adjustment was moderate, with a high rank correlation of .933.

Construct validity

Laminectomy rate loads on factor 1, and is related to all other area utilization indicators.

Face validity

Percutaneous transluminal coronary angioplasty (PTCA) is performed on patients with coronary artery disease (CAD). Previous studies of small area variation have found substantial variation in PTCA rates, though most analyses have been performed on Medicare data. ⁴⁸⁶

The clinical benefit of PTCA in many patients with unstable or chronic angina and recent myocardial infarction (MI) is unclear. No randomized controlled trials have demonstrated that PTCA improves clinical outcomes in many patients who commonly receive the procedure, and previous studies have documented large differences across hospitals in the likelihood of treatment with PTCA after MI and in other clinical settings.

Precision

Because adult admissions for PTCA are relatively common, it should be possible to generate precise estimates of utilization at the area level. However, random variation in utilization rates may become more problematic for relatively small areas (e.g., zip codes) or underpopulated areas (e.g., rural counties).

Minimum bias

Utilization rates standardized at the area level (e.g., adult population of a county or SMSA) may be biased by differences in the prevalence of coronary artery disease. The prevalence of CAD may, in turn, be related to the age structure of the population and the prevalence of behavioral or physiologic risk factors such as smoking and hyperlipidemia. Many previous studies have also shown that sociodemographic differences affect PTCA utilization, e.g., black patients are significantly less likely to undergo PTCA than non-black patients, particularly when indications were deemed equivocal. Little evidence exists on the extent to which area differences in socioeconomic and clinical characteristics may explain area differences in PTCA rates, though the large variations in rates across small geographic areas suggest that population characteristics are unlikely to explain most of the differences. ⁴⁸⁷ A second potential source of bias is the performance of procedures on an outpatient basis, which would not be captured in HCUP data and thus would not be reflected in HCUP-based area rate measures. However, less than 10% of PTCAs were performed on an outpatient basis in recent years, and so outpatient procedures do not appear to be quantitatively important enough to explain much of the observed variation. ²⁹⁷

Construct validity

For this indicator to perform well in identifying true quality of care problems, there must be evidence of significant inappropriate utilization in population-based studies. In addition, there must be substantial variation in the rate of inappropriate utilization across providers or small areas. Previous studies have classified possible indications for PTCA as appropriate, uncertain, or inappropriate, as a method for assessing quality of care.

In a study of seven of eight Swedish public heart centers, 38.3% of all PTCA procedures were performed for inappropriate indications and 30% for uncertain indications. ⁴⁷⁸ Another study of cardiac catheterization labs in Maryland used three different sets of criteria (RAND, ACC/AHA, and RAS) to assess the appropriateness of PTCA in these labs. Inappropriate indications ranged from 22 to 49%. The rate of uncertain indications was 29%, according to RAND criteria, the only set that includes this rating. ⁴⁸⁷ In a follow-up study of a coronary angiography study conducted in New York, a panel of Duke University cardiologists reviewed 308 records for appropriateness. Rates of PTCA performed for both inappropriate and uncertain reasons increased. That for in inappropriate indications was raised to 12%, while the rate of procedures performed for uncertain indications became 27%. ⁴⁷⁶

Fosters true quality improvement

In an effort to raise public perception of its quality without actually doing so, service providers might attempt to game results, or take action that would significantly decrease the incidence of an indicator or increase provider risk for these cases. This leads to an inaccurate assessment of quality by the public. Ideally, gaming is avoided in practice, present only in theory. However, providers might engage in practices, such as miscoding cases or recruiting patient groups that are known to have increased risk of CAD, in order to achieve more favorable quality assessment results with regard to PTCA. Perhaps instead of serving as quality assessments, results of these appropriateness studies can serve as guidelines for difficult clinical decisions. Patients and their physician might use them to spark questions and discussion about CAD, the patient's specific indications, and the most appropriate treatment, one that poses the least risk to the patient. ⁴⁸⁸ Implementing the results of these studies on an individual level might relieve undue stress of providers and reduce the temptation to inaccurately depict provider quality.

Prior use

The area-based rate of PTCA is a current indicator in the Dartmouth Atlas. ⁴⁷⁹

Precision

This indicator is precise, with a raw area level mean of 190.8 per 100,000 population and a standard deviation of 455.6. The systematic area level standard deviation is very high, at 0.28%. The area level variation also accounts for a very high percentage of total variation, at 0.21%. This means that relative to other indicators, a higher percentage of the variation occurs at the area level, rather than the discharge level. The signal ratio is also very high, at 97.3%. This means that it is very likely that the observed differences in area performance represent true differences in area performance. The very high R-square represents the high proportion of signal that can be extracted using multivariate techniques. However, multivariate methods have no additional impact, due to the already very high signal ratio.

Bias

Signal variance does not change substantially with risk adjustment. The indicator performs poorly on most measures of minimum bias. Risk adjustment impacts the absolute performance of areas moderately. In addition, 35.5% of areas changed more than 2 deciles. Risk adjustment appears to impact the highest decile disproportionately to the lowest decile, with 36.4 percent of areas in the highest decile remaining after risk adjustment, compared to 95.5% in the lowest decile. The overall relative impact of risk adjustment was also substantial, with a low rank correlation of .671.

Construct validity

PTCA rate loads on factor 1, and is related to all other area utilization indicators.

Face validity

Dehydration is a serious acute condition that occurs in frail patients and patients with other underlying illnesses, following insufficient attention and support for fluid intake. It is treatable with oral rehydration therapy and/or IV fluids. If left untreated in older adults, serious complications including mortality (over 50%) is very high. ⁵⁰⁶ Thus, prevention and effective management of dehydration may lead to fewer complications associated with severe dehydration, including hospitalization.

Precision

Dehydration is a somewhat common cause of hospital admission. We found little evidence on the precision of this indicator. One study did note that dehydration accounted for 7.3% of total admissions for ACSCs. ²⁸²

Minimum bias

We found no literature on the potential bias of this indicator. It is possible that the age structure of the population may affect admission rates for this condition, as the elderly and very young are more susceptible to dehydration. Socioeconomic factors may also affect admission rates, though we found no specific evidence confirming this hypothesis. We found no evidence on how comorbidities or other risk factors that may vary systematically by area influence hospitalization rates for dehydration. Finally, different thresholds for admission of patients with dehydration, rather than differences in quality of care, may also lead to area rate differences.

Construct validity

We found little literature on admission for dehydration as an ambulatory care sensitive condition indicator. Dehydration was originally included in John Billings' ⁴⁸⁹ set of indicators developed for the United Hospital Fund of New York. This set was developed by a physician panel. Evidence on sets of ambulatory care sensitive condition indicators are summarized at the beginning of this section, and should be referred to for this indicator.

Two studies of ACSC indicators reported validation work for dehydration independent of measure sets. Millman et al. ⁴⁹¹ reported that low-income zip codes had 2.1 times more dehydration hospitalizations per capita than high-income zip codes in 11 states in 1988. Billings et al. ⁴⁸⁹ found that low-income zip codes in New York City (where at least 60% of households earned less than $15,000 in 1988, based on adjusted 1980 Census data) had 2.0 times more dehydration hospitalizations per capita than high-income zip codes (where less than 17.5% of households earned less than $15,000). Household income explained 42% of the variation in dehydration hospitalization rates at the zip code level.

Fosters true quality improvement

We found no evidence of the impact such a measure would have on quality. As some dehydration can be managed on an outpatient basis, it is possible that a shift to outpatient care may occur. It is unknown whether hospitalizations for dehydration are appropriate, and thus whether the shift to outpatient care would be appropriate.

Prior use

This indicator was included in Billings set of Ambulatory Care Sensitive Conditions, developed in conjunction with the United Hospital Fund of New York. ⁴⁸⁹

Precision

This indicator is precise, with a raw area level rate of 139.9 per 100,000 population and a standard deviation of 103.2. The systematic area level standard deviation is moderate, at 0.04%. The area level variation also accounts for a moderate percentage of total variation, at 0.02%. The signal ratio is high, at 88.5%. This means that it is likely that the observed differences in area performance represent true differences in area performance. The high R-square represents the relatively high proportion of signal that can be extracted using multivariate techniques, though this is less than for other indicators. Multivariate techniques appear to have little additional impact.

Bias

Signal variance does not change substantially with risk adjustment. The indicator performs well on multiple measures of minimum bias, using age-sex adjustment. The rank correlation is very high at .957. Risk adjustment does not affect lowest decile disproportionately to the highest decile. In addition, few providers change more than 2 deciles with risk adjustment (4.6%). The absolute magnitude of the impact is minimal, with provider performance changing an average 9.2% relative to the mean with risk adjustment.

Construct validity

Dehydration is related to most other ACSC indicators.

Face validity

Bacterial pneumonia is a relatively common acute condition, treatable for the most part with antibiotics. If left untreated in a susceptible individual, pneumonia can lead to death (see pneumonia in-hospital mortality indicator). The elderly are particularly susceptible to pneumonia. A vaccine has been developed, which is used primarily in this population. This vaccine has been shown to be 45% effective in preventing hospitalizations in the elderly during peak seasons. ⁵⁰⁷ A 1995 survey of Americans older than 65 years found that only 35.6% of respondents reported ever receiving the vaccine. The same study found that minority populations reported a lower rate of vaccination (Hispanic, 24%, Black, 20%) as compared to Whites (37%). Overall state rates were not associated with minority rates of vaccination. This study is limited by the use of self-report, and therefore, vaccination may be underreported. ⁵⁰⁸

Appropriateness of admissions for bacterial pneumonia may account for some variation in admission rates. One study of emergency department triage strategies found a relatively modest impact on hospitalization rates for low risk cases - cases which may have been treatable on an outpatient basis. ⁵⁰⁹

Precision

Pneumonia is a very common cause of hospital admission, particularly in the elderly, suggesting that relatively precise estimates of area rates should be feasible. However, little evidence exists on the precision or variation in pneumonia admission rates.

Minimum bias

We found no literature on the potential bias of this indicator. As the elderly are more susceptible to pneumonia, rates may be associated with the age structure of the population. Rates may also differ with comorbid diseases and socioeconomic status. Immunosuppressed patients are more likely to both develop and require hospitalization as a result of pneumonia. Some causes of immunosupression may vary systematically by area. However, we found no evidence that comorbidities or other risk factors that may vary systematically by area significantly affect the incidence of hospitalization for pneumonia. Physician thresholds for admitting patients with pneumonia also differ, which may contribute to observed differences in admission rates.

Construct validity

We found little literature on admission for pneumonia as an ambulatory care sensitive condition indicator. Pneumonia was originally included in Weissman's ²⁷⁶ set of avoidable hospitalization indicators. This set was developed by physician panels. Evidence on sets of ambulatory care sensitive condition indicators are summarized at the beginning of this section, and should be referred to for this indicator.

Two studies of ACSC conditions have examined pneumonia independently. Millman et al. ⁴⁹¹ reported that low-income zip codes had 5.4 times more pneumonia hospitalizations per capita than high-income zip codes in 11 states in 1988. Billings et al. ⁴⁸⁹ found that low-income zip codes in New York City (where at least 60% of households earned less than $15,000 in 1988, based on adjusted 1980 Census data) had 5.4 times more pneumonia hospitalizations per capita than high-income zip codes (where less than 17.5% of households earned less than $15,000). Household income explained 53% of the variation in pneumonia hospitalization rates at the zip code level.

Fosters true quality improvement

We found no evidence of the impact such a measure would have on quality. As some cases of pneumonia can be managed on an outpatient basis, it is possible that use of this indicator may encourage a shift to outpatient care without actually affecting patient outcomes. Such a shift might be inappropriate for more severely ill patients.

Prior use

This indicator was included in Weissman's set of avoidable hospitalizations. ²⁷⁶ Immunization preventable pneumonia was included in the HCUP I indicator set.

Precision

This indicator is precise, with a raw area level rate of 395.6 per 100,000 population and a standard deviation of 208.5. The systematic area level standard deviation is high, at 0.09%. The area level variation also accounts for a high percentage of total variation, at 0.03%. The signal ratio is very high, at 92.9%. This means that it is likely that the observed differences in area performance represent true differences in area performance. The very high R-square reflects the large proportion of signal that is extractable using multivariate methods. However, multivariate techniques have little additional impact, due primarily to the already large signal ratio.

Bias

Signal variance does not change significantly with risk adjustment. The indicator performs well on multiple measures of minimum bias. The rank correlation is good at 0.923. Risk adjustment appears to affect the highest decile disproportionately to the lowest decile, as 68.2% of areas in the highest decile and 90.9% of the lowest decile remain after risk adjustment. However, this is consistent with other indicators. The indicator still performs well relative to other indicators on this measure. The absolute magnitude of the impact is moderate, as is the relative impact.

Construct validity

Bacterial pneumonia is related to most other ACSC indicators.

Face validity

Urinary infection (UTI) is a common acute condition. Uncomplicated urinary tract infections are treatable with antibiotics on an outpatient basis. If left untreated or incompletely treated in a susceptible individual, urinary tract infections can spread into the kidneys (pyelonephritis) or develop into septicemia. Among children, admission for UTI is associated with physiological abnormalities and is rare.

Precision

We found little evidence on the precision of this indicator. One study did note that UTIs and kidney infections accounted for 10.6% of total admissions for ACSCs. ²⁸²

Minimum bias

Urinary tract infections are a somewhat common cause of hospitalization. We found no literature on the potential bias of this indicator. We found no evidence that comorbidities or other risk factors that may vary systematically by area may increase the incidence of hospitalization for UTI. Thresholds for admission of patients with urinary tract infections may differ across areas, potentially contributing to observed differences in area hospitalization rates.

Construct validity

We found little literature on admission for urinary infection as an ambulatory care sensitive condition indicator. UTI was originally included in both John Billings' ⁴⁸⁹ set of indicators developed for the United Hospital Fund of New York, and in Weissman's ²⁷⁶ set of indicators. These sets were developed by physician panels. Evidence on sets of ambulatory care sensitive condition indicators are summarized at the beginning of this section, and should be referred to for this indicator.

Two studies of ACSC indicators reported validation work for UTI independent of measure sets. Millman et al. ⁴⁹¹ reported that low-income zip codes had 2.8 times more UTI hospitalizations per capita than high-income zip codes in 11 states in 1988. Billings et al. ⁴⁸⁹ found that low-income zip codes in New York City (where at least 60% of households earned less than $15,000 in 1988, based on adjusted 1980 Census data) had 2.2 times more UTI hospitalizations per capita than high-income zip codes (where less than 17.5% of households earned less than $15,000). Household income explained 28% of the variation in UTI hospitalization rates at the zip code level.

Fosters true quality improvement

We found no evidence of the impact such a measure would have on quality. As most UTIs can be managed on an outpatient basis, it is possible that a shift to outpatient care may occur. The "appropriate" rate of inpatient treatment for UTI is unknown, and thus there is little evidence on whether the shift to outpatient care would reduce quality of care.

Prior use

This indicator was included in Billings ⁴⁸⁹ set of Ambulatory Care Sensitive Conditions, developed in conjunction with the United Hospital Fund of New York, and in Weissman's set of avoidable hospitalizations. ²⁷⁶

Precision

This indicator is precise, with a raw area level rate of 145.1 per 100,000 population and a standard deviation of 89.5. The systematic area level standard deviation is moderate, at 0.04%. The area level variation also accounts for a high percentage of total variation, at 0.01%. This means that relative to other indicators a higher percentage of the total variation reflects unobserved patient differences. The signal ratio is high, at 84.9%. This means that it is likely that the observed differences in area performance represent true differences in area performance, though this is lower than other indicators. The high R-square reflects the large proportion of signal variance that can be extracted using multivariate techniques, though lower than some other indicators. Multivariate techniques have little additional impact.

Bias

Signal variance does not change with risk adjustment. The indicator performs well on multiple measures of minimum bias. The rank correlation is good at 0.914. Risk adjustment appears to impact the highest decile disproportionately to the lowest decile, as 68.2% of areas in the highest decile and 90.9% of the lowest decile remain after risk adjustment. The absolute magnitude of the impact is moderate, as is the relative impact.

Construct validity

Urinary infection is related to most other ACSC indicators.

Face validity

Perforated appendix may occur when appropriate treatment for acute appendicitis is delayed for any variety of reasons. Such delay could result from access to care problems, patients failing to interpret symptoms as potentially important, misdiagnosis and other delays in obtaining surgery.

Precision

Perforated appendix is a relatively common condition, occurring in ¼ to 1/3 of hospitalized acute appendicitis patients. ⁵¹⁰ Thus, it is likely to be measured with good precision.

Minimum bias

Observational studies utilizing large administrative databases have noted higher rates of perforated appendix in males, ⁵¹⁰ patients with mental illness or substance abuse disorders, ⁵¹⁰ diabetics, ⁵¹⁰ blacks, ⁵¹⁰ and children under the age of four (though appendicitis is rare in this age group). ⁵¹¹ If areas have an unusually high proportion of patients with diabetic comorbidity, or patient with substance abuse or psychiatric comorbidity may have higher rates of perforated appendix that may not be due to actual differences in quality of care. However, since appendicitis is relatively rare as compared with these comorbidities, it is unlikely that an area rate would change significantly due to perforated appendix in these populations.

Construct validity

Three recent studies have examined perforated appendix as a measure of access to care. The first examined all California adult admissions (18-64 years of age), with acute appendicitis (96,587 cases). ⁵¹⁰ They found that patients with either no insurance or Medicaid had just under 50% greater risk of perforated appendix than HMO covered patients. They interpreted this result to be evidence of potential access to care problems. They also found a 20% increased risk in patients with private fee-for-service insurance. In a follow-up to this study, Blumberg et al. noted that when examining the high rate of perforated appendix in the black population at Kaiser, patients with almost identically comprehensive coverage, black patients did not have higher rates of delayed admissions, a possible indication of poor quality care. They postulated that rather, delay in seeking care may explain some of the differences observed. ⁵¹² Thus, it is unclear whether the higher rate of perforated appendix is due to quality of care problems, perceived access to care difficulties, or other reasons for the delay in seeking care.

The second study took a similar approach in analyzing pediatric admissions for acute appendicitis in Washington state. ⁵¹¹ The rate of perforated appendix was again increased in Medicaid patients [Adj. OR 1.3, 95% CI (1.2-1.4)]. Another study in a pediatric population examined reasons for delay to surgery and insurance status in a New York pediatric population through retrospective chart review. They noted that Medicaid or uninsured children had both a higher perforation rate and a longer duration of symptoms before presenting to a health care professional as compared to HMO or private fee for service insured children. There were no differences between the types of insurance in the time to surgery after presentation. ⁵¹³ Unfortunately the authors did not analyze how much of the variance in perforated appendix could be explained by delays in seeking care.

Weissman et al. their analysis of avoidable hospitalizations found that uninsured had a relative risk of 1.14-1.20 of admission for ruptured appendix after adjusting for age and sex. Medicaid patients had a relative risk of .45-.58, suggesting that in at least this case, Medicaid patients are not at increased risk for ruptured appendix. ²⁷⁶

Fosters true quality improvement

Rates of perforated appendix could be increased by increasing the denominator, total patients admitted for appendicitis, by increasing the number of patients with borderline or questionable cases undergoing appendectomy. ⁵¹⁰

Prior use

Perforated appendix was included in both the previous HCUP I indicator set, as well as in Weissman's set of avoidable hospitalizations. ²⁷⁶

Precision

This indicator is precise, with a raw area level rate of 33.3% and a substantial standard deviation of 14.4%. The systematic area level standard deviation is very high, at 2.75%. The area level variation also accounts for a very high percentage of total variation, at 0.34%. This means that relative to other indicators, a higher percentage of the variation occurs at the area level, rather than the discharge level. However, the signal ratio is low, at 26.5%. This means that it is likely that much of the observed differences in area performance do not reflect true differences in performance. The R-Square does demonstrate the improvement in the amount of signal that can be extracted using multivariate techniques. Nonetheless, the R-square is still low at 39.4%.

Bias

Signal variance decreases by over 15% with risk adjustment by age and sex. This indicates that some of the apparent signal is due to systematic differences in patient demographics. The demographic (age-sex) adjustment had minimal impact on most measures of minimum bias. The rank correlation was high at 0.969. The number of areas changing more than two deciles was relatively low, at 2.5%. Relative to other indicators, this risk adjustment had a moderate impact on the lowest end of the distribution, with 85% of providers in the lowest decile remaining after risk adjustment.

Construct validity

Perforated appendix is negatively related to the other ACSC conditions. It is positively related to low birth weight.

Face validity

Both stable and unstable angina are symptoms of potential coronary artery disease. Stable angina can be managed in an outpatient setting, using drugs such as aspirin, beta blockers, and advise to change diet and exercise habits. ⁵¹⁴ Unstable angina may be managed on an outpatient basis, but admission is required for more severe cases, such as patients with recurrent or progressive symptoms. ⁵¹⁵ Effective management of coronary disease reduces the occurrence of major cardiac events such as heart attacks, and may also reduce admission rates for angina.

Precision

Unstable angina is a very common reason for hospital admission. One study noted that angina accounted for 16.3% of total admissions for ACSCs. ²⁸² Thus, reasonably precise estimates of area angina rates should be feasible.

Minimum bias

We found no literature on the potential bias of this indicator. The incidence of angina is related to the incidence of coronary artery disease (CAD), which is in turn related to the age structure and risk factors (smoking, hyperlipidemia, hypertension, diabetes), in a population. Some areas may systematically vary in the incidence of angina, as a result of these factors and potentially correlated differences in socioeconomic status. Elderly age (over 70), diabetes and hypertension have also been associated with higher risk angina. ⁵¹⁵ Finally, physicians may differ in their thresholds for admitting patients with less stable angina. Little evidence exists on the extent to which these factors may account for area differences in angina admission rates.

Construct validity

We found little literature on admission for angina as an ambulatory care sensitive condition indicator. Angina was originally included in John Billings' ²⁷⁵ set of indicators developed for the United Hospital Fund of New York. This set was developed by a physician panel. Evidence on sets of ambulatory care sensitive condition indicators are summarized at the beginning of this section, and should be referred to for this indicator.

Two studies of ACSC indicators reported validation work for angina independent of measure sets. Millman et al. ⁴⁹¹ reported that low-income zip codes had 2.7 times more angina hospitalizations per capita than high-income zip codes in 11 states in 1988. Billings et al. ⁴⁸⁹ found that low-income zip codes in New York City (where at least 60% of households earned less than $15,000 in 1988, based on adjusted 1980 Census data) had 2.3 times more angina hospitalizations per capita than high-income zip codes (where less than 17.5% of households earned less than $15,000). Household income explained 13% of the variation in angina hospitalization rates at the zip code level.

Fosters true quality improvement

We found no evidence of the impact such a measure would have on quality. As some angina can be managed on an outpatient basis, it is possible that a shift to outpatient care may occur without true changes in the occurrence of angina complications. However, it seems unlikely that severe angina, requiring observation, would be shifted to outpatient settings.

Prior use

This indicator was included in Billings set of Ambulatory Care Sensitive Conditions, developed in conjunction with the United Hospital Fund of New York. ²⁷⁵

Precision

This indicator is precise, with a raw area level rate of 166 per 100,000 population and a standard deviation of 135.7. The systematic area level standard deviation is high, at 0.06%. The area level variation also accounts for a high percentage of total variation, at 0.04%. The signal ratio is very high, at 91.6%. This means that it is likely that the observed differences in area performance represent true differences in area performance. The very high R-square reflects the large proportion of signal that can be extracted using multivariate techniques. Such techniques do not have substantial additional impact, primarily due to the already very high signal ratio.

Bias

Signal variance does not change with risk adjustment. The indicator performs well on multiple measures of minimum bias. The rank correlation is very good at 0.968. Risk adjustment appears to disproportionately impact the high decile compared to the low decile, as 63.6% of areas in the highest decile and 90.9% of the lowest decile remain after risk adjustment. Few providers change more than two deciles. The absolute magnitude of the impact is moderate, with an average change with risk adjustment (relative to the mean) of 10.6%.

Construct validity

Angina is related to most other ACSC indicators.

Face validity

Asthma is one of the most common chronic diseases in the United States, affecting nearly 15 million individuals. ⁵¹⁶ In 1997, there were about 433,000 ⁵¹⁷ to 484,000 hospitalizations for asthma in the US, ⁵¹⁸ of which at least 252,000 involved persons 18 or more years of age. In 1996, asthma was the 10th most common principal diagnosis in emergency department (ED) visits and the ninth most common diagnosis in hospital outpatient departments. ⁵¹⁹ In this discussion, we will consider only admission for asthma in adults. A separate discussion of the evidence for the pediatric asthma indicator (indicator 31) is included elsewhere in this report.

There is widespread consensus, embodied in the National Asthma Education Program, ⁵²⁰ that asthma is a readily treatable chronic disease that can be managed effectively in the outpatient setting. As Healthy People 2010 indicates, "effective management of asthma includes four components: avoiding or controlling the factors that may make asthma worse..., taking appropriate medications tailored to the severity of the disease, objective monitoring of the disease by the patient and the health care professional, and actively involving the patient in managing the disease."

Many asthma exacerbations are preventable using inhaled anti-inflammatories such as corticosteriods and mast cell stabilizers, or treatable in the outpatient setting using beta agonists and systemic corticosteroids. Observational studies offer limited evidence that inhaled steriods may decrease risk of admission by up to 50%.^{521, 522} Potential confounding factors such as asthma severity, however, limit the conclusiveness of these results. In addition, factors that explain variation in the risk of hospitalization at the patient level may not explain variation in area-level hospitalization rates.

Precision

Asthma is a common cause of admission for adults, and as such this measure is likely to have adequate precision. Blustein et al. noted that asthma accounted for 5% of ACSC admissions for Medicare beneficiaries. ²⁸² One United Kingdom study noted that 51% of health authorities changed over 10 or more places when ranked by asthma admission rates over a 1 year period (1993-1994). ⁵²² The generalizability of this study to the United States is unknown.

Minimum bias

Numerous environmental risk factors for asthma have been identified, and some of these factors are more prevalent in certain communities than in others. Indoor allergens such as cockroaches and dust mites may be more common in lower-income areas, and are probably associated with increased frequency and severity of asthma symptoms. ⁵²³ Tobacco smoke is the most important indoor irritant and is a major precipitant of asthma symptoms in both children and adults.^524-527 Jindal and colleagues ⁵²⁶ found that exposure of adults to environmental tobacco smoke is associated with decreased pulmonary function, increased medication requirements, and more frequent absences from work. Outdoor air pollution, especially respirable particulates, may also play a role.^{520, 528-532} In addition, ozone and SO₂ have been associated with increased emergency department visits and hospitalizations rates.^531-538 Increasing air pollution has been specifically correlated with higher admission rates in London (Ozone, NO₂, SO₂, and black smoke), ⁵³⁹ and Seattle (ambient air pollution) ⁵⁴⁰ .

Occupation is a potentially important source of bias in comparing asthma hospitalization rates among adults. A recent review of 43 studies from 19 different countries found that at least 9%, and perhaps as much as 15% (based on the highest quality studies), of the population burden of asthma is attributable to occupational factors. ⁵⁴¹ About 250 agents capable of causing occupational asthma have been identified. ⁵⁴² These agents affect individuals in specific occupations, such as carpenters and woodworkers, ⁵⁴³ who may cluster in certain geographic areas.

Race represents one of the most complex potentially biasing factors for this indicator. Black patients have consistently been shown to have higher asthma admission rates,^544-546 even when stratifying for income and age. ⁵⁴⁷ One study examining differences in asthma health care utilization noted that African Americans made fewer asthma-related primary care and specialist visits than Caucasian patients (47.6% vs. 70.2% and 27% vs. 38.8%). There were no differences in hospitalization rates by race, but African-American patients had lower household incomes, and made more emergency department visits (proxy for either access to care or asthma severity). ⁵⁴⁸ Similarly, Hispanics have been shown to have higher admission rates than non-Hispanic whites (or areas with higher percentages of Hispanics have been shown to have higher admission rates), although none of thesestudies controls for SES.^{545, 549, 550} To the extent that true differences in disease prevalence or severity are responsible for racial variation in hospitalization rates, race should be adjusted for in comparing asthma hospitalization rates across areas. On the other hand, to the extent that minority patients have less access to care or poorer quality of outpatient care, race should not be adjusted for.

Construct validity

Little evidence has been reported conclusively attaching poor quality of care to higher area admission rates. However, numerous studies have shown that asthma hospitalization rates are associated with socioeconomic factors, including median household income (at the area level) and lack of insurance (at the individual level). A study of asthma hospitalization rates in California in 1993 (ages 0-64) found that areas with median household incomes under $35,000 had hospitalization rates that were 1.5 times higher than areas with higher median incomes. ⁵⁴⁷ In Boston, in 1992, age and gender standardized hospitalization rates (all ages) were correlated with percentage poverty in an area (r=0.68), percentage holding a bachelor's degree (r=-0.61), and income (r=-0.51). ⁵⁵⁰ Within New York City in 1994, asthma hospitalization rates were negatively correlated with a zip code area's median household income (r=-0.67), and positively correlated with the percentage of minorities in the population (r=0.82). ⁵⁴⁹ These findings confirm an earlier study by Billings et al., ⁴⁸⁹ who reported 6.4-fold variation in asthma hospitalization rates at the zip code level in New York City in 1988, with 70% of this variation explainable by the percentage of households with annual income below $15,000. Millman et al. ⁴⁹¹ reported that low-income zip codes had 5.8 times more asthma hospitalizations per capita than high-income zip codes in 11 states in 1988. Using New York State data, Lin et al showed that hospitalization rates were higher in areas with higher poverty, unemployment, minority populations, and lower education levels. ⁵⁴⁵ Even in England, 45% of the variation in asthma hospitalization rates across 90 family health services authorities in 1990-95 was attributable to socioeconomic factors, plus the availability of secondary care. ⁵⁵¹ To our knowledge, only one study has reported partial correlations; ⁵⁵² it found that that in New York City, the percentage of African-American residents was the strongest predictor, and median household income was the next strongest predictor, of asthma hospitalization rates.

Similar findings have been demonstrated at the individual level. For example, Weissman et al. ²⁷⁶ noted that adjusted relative rate of adult asthma hospitalization for uninsured persons, compared with privately insured persons, was 1.42 (95% CI, 1.20-1.63) in Massachusetts and 1.19 in Maryland (95% CI, 0.92-1.45). The comparable rates for Medicaid beneficiaries were 1.84 (95% CI, 1.63-2.05) and 1.61 (95% CI, 1.33-1.90), respectively. Bierman et al. ⁵⁵³ noted that the asthma hospitalization rate among persons with private insurance nationwide was 33% lower, and that among Medicaid beneficiaries was significantly higher, than that in the general population.

The observation that asthma admission rates are higher in areas with low SES has led some researchers to hypothesize that lack of access to care, or poor quality outpatient care, may lead to higher admission rates. Bindman et al. ²⁸⁴ showed that asthma hospitalization rates across 41 sampled areas in California were significantly correlated (r=0.47) with self-rated access to needed medical care, according to community telephone surveys. Although analyses of the National Health and Nutrition Examination Survey found that Medicaid enrollment and Spanish language preference were associated with inadequate asthma therapy, these deficiencies in care were not directly linked to hospitalizations. ⁵⁵⁴ Studies from other settings have shown that African-American asthmatics tend to have fewer scheduled primary care visits, and more hospitalizations and emergency room visits, than White asthmatics.^{555, 556} African-Americans' use of asthma medications may also be less consistent with current practice guidelines. ⁵⁵⁷

Some weak evidence indicates that patients admitted to the hospital often receive suboptimal outpatient care, as measured by guideline adherence. The National Asthma Education Program (NAEP) Guidelines suggest that patients receive proper asthma education, including use of an MDI and written action plans in case of exacerbation. ⁵²⁰ One study of patients hospitalized in an inner-city teaching hospital noted that only 28% had received an action plan, and 11% could not demonstrate proper use of an MDI, despite reporting having been shown previously by health care personnel. In addition, 69 of the 101 patients reported were prescribed theophylline without first being prescribed anti-inflammatory inhalers. Finally, 60% of the patients who contacted physicians during the current exacerbation reported that the health care provider made no changes in treatment. ⁵⁵⁸ Similar results, with 95% of patients failing to use an action plan and 51% having inadequate knowledge, were reported from an Australian hospital. ⁵⁵⁹ Neither of these studies had a control group with non-hospitalized asthmatics, and both relied on self-reported treatment data.

Few studies have directly linked high-quality processes of outpatient care with lower hospitalization rates at either the area or the individual level. An in-depth study of asthma treatment practices in New Haven, Boston, and Rochester found that the community with the highest asthma hospitalization rate (Boston) also had lower use of inhaled anti-inflammatory agents and oral steroids. The threshold for admission also appeared to be lower in Boston, as fewer of the admitted patients were hypoxemic, relative to the other cities. ⁵⁶⁰ One case control study from a large health maintenance organization established that not having a written asthma management plan was a strong risk factor for asthma hospitalization (after adjusting for severity of asthma), but the use of antiinflammatory medications was not. ⁵⁶¹ Although these studies focused on children rather than adults, the results provide limited support for theconstruct validity of the asthma hospitalization rate as an indicator of access to high-quality outpatient care.

Fosters true quality improvement

We located no studies discussing the ability of this indicator to foster true quality improvement. One study from the United Kingdom ⁵⁵¹ argues that area admission rates for asthma are not a good measure of quality, due to the confounding factors discussed above. While we located no studies examining the potential for gaming for this indicator, it is possible that patients who present to outpatient clinics or emergency rooms as candidates for admission would not be admitted, but rather treated in the outpatient or ER setting. There is little evidence currently to suggest that asthmatics are being inappropriately denied admission to the hospital, although this problem could emerge in the future.

Prior use

Most published indicators of asthma admission include pediatric as well as adult patients in a single indicator, though they are separate indicators in this report. Children under the age of 5 are sometimes excluded. Admission for asthma has been implicated as an ambulatory care sensitive condition, included in both the preventable hospitalization set of Weissman et al. ²⁷⁶ and the set of ACS conditions developed by Billings et al., ⁴⁸⁹ in conjunction with the United Hospital Fund of New York. The UK National Health Service has designated asthma admission as a High Level Performance Indicator. In addition, Healthy People 2010 has set a goal to reduce the nationwide asthma admission rate from 12.5 (in 1998) to 7.7 per 10,000 children and adults aged 5-65 years, and from 17.7 to 11 per 10,000 adults over 65 years of age. ⁵

Precision

This indicator is adequately precise, with a raw area level rate of 107.9 per 100,000 population and a standard deviation of 81.7. The systematic area level standard deviation is moderate, at 0.04%. The area level variation also accounts for a moderate percentage of total variation, at 0.02%. These suggests that relative to other indicators a lower proportion of variance occurs at the area level, rather than the discharge level. The signal ratio is high, at 83.6%. This means that it is likely that the observed differences in area performance represent true differences in area performance, though some reflects unsystematic differences. The high R-square reflects the large proportion of signal that can be extracted using multivariate techniques, though lower than other indicators. Such techniques do not have substantial additional impact.

Bias

Signal variance does not change with risk adjustment. The indicator performs well on multiple measures of minimum bias. The rank correlation is very good at 0.989. Risk adjustment does not appear to impact the extremes of the distribution substantially. No providers change more than two relative deciles. The absolute magnitude of the impact is minimal, with an average change with risk adjustment (relative to the mean) of 3.8%.

Construct validity

Adult asthma is related to most other ACSC indicators.

Face validity

The incidence of COPD has been increasing in the last decade, ⁵⁶² as have admissions for COPD. Admissions for COPD include exacerabations of COPD, respiratory failure, and rarely lung volume reduction surgery or lung transplantation. COPD accounts for over $20 billion in health care expenditures. Data suggests that more than 2/3 of these costs are incurred by only 10% of persons with COPD. ⁵⁶³

Practice guidelines for COPD have been set forth and published over the past decade. ⁵⁶⁴ Based on consensus, the three major guidelines (European Respiratory Society, American Thoracic Society and British Thoracic Society) agree that appropriate care for COPD includes spirometry, medication, and monitoring. All also agree that advice to quit smoking is a critical intervention. Specific recommendations for the indications of drug use vary depending on the nature of disease, though they often include bronchodialators (combined beta₂-agonist and anticholinergic), theophylline and corticosteriods. These guidelines are based on consensus statements, not empiric evaluation of the evidence, and call for the need for more research on the effectiveness of treatments. ⁵⁶⁵

With appropriate outpatient treatment and compliance, hospitalizations for the exacerbations of COPD and decline in lung function should be minimized. COPD was selected by a physician panel as an ambulatory care sensitive condition. ²⁷⁵

Precision

COPD is a common disease that accounts for a substantial number of hospital admissions (>13% of admissions), ⁵⁶⁶ suggesting that reasonably precise area rate estimates are feasible. We were unable to identify studies that examined geographic variation in admission for COPD. Slight seasonal variations have been noted in younger age groups, though in patients over 65 no seasonal variations in hospital admissions have been noted. ⁵⁶⁷ Billings' original study from New York reported only 1.8-fold variation in COPD hospitalization rates, with a coefficient of variation of 0.742. ⁴⁸⁹

Minimum bias

Exacerbations of COPD and the rate of pulmonary function decline can be affected by patient characteristics. Cigarette smoking is a leading cause of COPD. Smoking after the development of COPD has been shown to accelerate the rate of pulmonary decline, with quitters sustaining less pulmonary decline that continuous smokers. ⁵⁶⁸ One study showed a small but significant increase in pulmonary decline in patients that quit smoking then restarted, compared with patients that never quit. ⁵⁶⁹ Thus, patient compliance is an important determinant of COPD admission rates. Actual smoking cessation by COPD patients is less than optimal, with many patients failing to comply despite ambulatory interventions. Even with intensive smoking cessation interventions, continuous quit rates remain at around 20%. ⁵⁶⁸ Lower rates have been reported in other studies with different patient populations, suggesting that patient characteristics may influence responsiveness to treatment.

Other factors have been associated with increased hospitalizations for COPD. For example, the association between lower socio-economic status and increased likelihood of hospital admission ⁵⁷⁰ may be due to higher occupational exposure to harmful substances, smoking rates, and infection rates postulated to exist among lower socio-economic groups. ⁵⁷⁰ However, Billings' original study from New York reported only 3% of variance explained by household income. ⁴⁸⁹ In addition, COPD is a progressive disease, and disease severity varies considerably across patients and over time. As lung function declines with time in COPD, older patients have higher rates of pulmonary decline, but heavier smoking and genetic factors also influence the rate of decline.

One environmental factor, daily increases in air pollution, has been associated with increased daily rates of COPD hospitalization.^571-579 The robustness of the association, and the type and amount of air pollution affecting admissions, varies between study. Few studies have examined air pollution differences between geographical areas and its association with admission for COPD.

Thus, in addition to direct measures of disease severity, smoking status, age, and socio-economic factors may increase the likelihood of admission for COPD. These factors are candidates for use in risk-adjustment models.

One study did indicate that COPD in younger populations might include some miscoding for acute bronchitis. ⁵⁶⁷ It is unknown whether such coding differences lead to biases in area-level estimates of COPD admission rates.

Construct validity

The extent to which the admission rate for COPD relates to the quality of the outpatient health care provided has not been widely evaluated, though some studies have examined readmission rates. Weissman et al. did not evaluate COPD, but Bindman et al. reported that self-reported access to care explained 27% of the variation in COPD hospitalization rates (e.g., less than for asthma, CHF, or diabetes) at the zip code cluster level. ²⁸⁴ Millman et al. ⁴⁹¹ reported that low-income zip codes had 5.8 times more COPD hospitalizations per capita than high-income zip codes in 11 states in 1988. However, Billings et al.'s ⁴⁸⁹ findings were weaker; low-income zip codes in New York City (where at least 60% of households earned less than $15,000 in 1988, based on adjusted 1980 Census data) had 1.8 times more COPD hospitalizations per capita than high-income zip codes (where less than 17.5% of households earned less than $15,000). Household income explained only 3% of the variation in COPD hospitalization rates at the zip code level.

Some articles discuss the adherence to practice guidelines and other therapies by both physicians and patients, though they do not generally examine the relationship between adherence rates and hospitalization rates for COPD. One study found varying rates of physician compliance with practices expected to improve quality for patients admitted for COPD. In this study, over 2/3 of physicians performed a complete history and physical examination and some nutritional screening. Almost all provided some sort of appropriate pharmacologic treatment (96%). However, few gave smoking cessation advice during the hospitalization (23%), and almost none provided appropriate discharge planning and education (.2%). ⁵⁸⁰ Another study found that often physicians did not adhere to practice guidelines. In this study, smoking cessation counseling was provided in 14.3% of ambulatory visits. Physicians prescribed medication contrary to indications. More than ¼ of visits resulted in a prescription of theophylline, though this drug is considered for use as a "step 3" drug, and 5% of visits resulted in a prescription for ipratropium, the first-line therapy. ⁵⁸¹ No information was provided regarding the patients' severity of disease.

As noted above, patient compliance also influences the effectiveness of therapy. One study found patient compliance with inhalers ranged from 40%-60%. Patients reported higher compliance than that found by weighing canisters. Both measures may overestimate compliance, according to the authors. Patient compliance decreased over the 5-year follow-up period. ⁵⁶⁸ Other studies have found similar rates of non-compliance. ⁵⁸¹

Fosters true quality improvement

One study examined COPD as one of three diseases possibly affected by access to care. Increased access to care in this randomized trial was associated with increased admission rates, possibly because of more detection of significant respiratory impairments in the community. ⁵⁸² Thus, higher rates of COPD admission may in part reflect improvements in access to care, rather than deficient ambulatory care per se. However, this finding may also reflect a decline in the threshold for admission of "marginal" COPD cases in areas with greater access to care. It is possible that changes in coding practices, for example, coding patients as acute patients or omitting COPD codes, may reduce observed rates of COPD admissions. Recent investigations by the Medicare program into coding practices involving respiratory disease admissions raise the possibility that some COPD admissions reflect "upcoding." If the measure were used as an indicator, a decline in COPD admission rates may simply reflect a reverse change in coding practices.

Prior use

This measure was originally developed by Billings and colleagues in conjunction with the Ambulatory Care Project of the United Hospital Fund of New York, ²⁷⁵ and was subsequently adopted by the Institute of Medicine. ⁵⁸³ It has been widely used in a variety of studies of avoidable or preventable hospitalizations. At least 6 states (MA, NE, UT, VA, MI, NY) are reportedly using this set of measures "as guidance for policy and as evaluation and decision aids." ⁵⁸⁴ Note that COPD was not among the conditions identified by Weissman et al in 1992 as "avoidable hospital conditions." ²⁷⁶ A related indicator, admission for asthma, COPD, or pneumonia among patients with a prior diagnosis of COPD, was recently recommended as a measure of access to care for elderly Medicare beneficiaries. ⁵⁸⁵

Precision

This indicator is very precise, with a raw area level rate of 324.0 per 100,000 and a standard deviation of 203.8. The systematic area level standard deviation is high, at 0.10%. The area level variation also accounts for a high percentage of total variation, at 0.05%. This means that relative to other indicators, a higher percentage of the variation occurs at the area level, rather than the discharge level. The signal ratio is very high, at 93.4%. This means that it is very likely that the observed differences in area performance represent true differences in area performance. The very high R-square reflects the large proportion of signal that can be extracted using multivariate techniques. Such techniques do not have substantial additional impact, primarily due to the already very high ratio.

Bias

Signal variance decreases by over 15% with risk adjustment, indicating that some of the true variation among providers is due to differences in patient demographic characteristics. The indicator performs well on multiple measures of minimum bias. The rank correlation is good at 0.933. Risk adjustment appears to affect the high decile disproportionately to the low decile, as 68.2% of areas in the highest decile and 86.4% of the lowest decile remain after risk adjustment. The absolute magnitude of the impact is moderate, as is the relative impact.

Construct validity

COPD is related to most other ACSC indicators.

Face validity

There are significant differences in physician management of patients with congestive heart failure, particularly depending on physician specialty (cardiologists vs. internists).^{586, 587} In the community- hospital setting, the clinical practices of cardiologists are more compatible with published treatment guidelines than the clinical practices of other physicians. The benefits of cardiology specialty care include lower CHF readmission rates and better post-discharge quality-of-life measures, rather than lower mortality rates, fewer hospital charges, or shorter length of stay. ⁵⁸⁸ Despite clinical trial evidence demonstrating the effectiveness of ACE inhibitors, the drug remains under-prescribed by most physicians. ⁵⁸⁹ There are significant differences in physician management of patients with congestive heart failure, particularly depending on physician specialty (cardiologists vs. internists).^{586, 587} Thus, it is plausible that such differences in community practices are reflected in differences in CHF admission rates. Because of the large numbers of patients with CHF and their substantial mortality, morbidity and cost of care, these differences may have a major impact on outcomes and health care costs.

Precision

Congestive heart failure is one of the leading diagnosis-related group (DRG) discharge diagnosis in the United States. ⁵⁸⁷ In the NIS, almost 3% of all discharges are for CHF, with roughly 200 discharges per hospital. Therefore, one can obtain relatively precise estimates of admission rates for CHF, although random variation may be important for small hospitals and rural areas. Billings' original study from New York reported up to 4.6-fold variation in CHF hospitalization rates, with a coefficient of variation of 0.646. ⁴⁸⁹

Minimum bias

Important determinants of patient outcomes with CHF include certain demographic variables (e.g., patient age), clinical measures (e.g., left ventricular ejection fraction and serum creatinine), management issues (e.g., documentation of left ventricular function and documentation of etiology of CHF), and treatment strategies (e.g., ancillary drug use). ⁵⁸⁹ These factors appear to be correlated with socioeconomic status. Billings' original study from New York found that 59% of the substantial cross-variance was associated with differences in household income. ⁴⁸⁹ Only limited evidence exists on the extent to which such factors, rather than access to and use of high-quality medical care, accounts for differences across areas.

Construct validity

Some evidence suggests that access to care influences CHF hospitalization rates. Weissman et al. found CHF hospitalization rates to be variably associated with lack of insurance (adjusted RR=1.17 and 1.81 in MA and MD, respectively) and Medicaid (adjusted RR=2.41 and 2.53 in MA and MD, respectively). ²⁷⁶ Bindman et al. reported that self-reported access to care explained 50% of the variation in CHF hospitalization rates (e.g., more than for any other condition) at the zip code cluster level. ²⁸⁴ Millman et al. ⁴⁹¹ reported that low-income zip codes had 6.1 times more CHF hospitalizations per capita than high-income zip codes in 11 states in 1988. Billings et al. ⁴⁸⁹ found that low-income zip codes in New York City (where at least 60% of households earned less than $15,000 in 1988, based on adjusted 1980 Census data) had 4.6 times more CHF hospitalizations per capita than high-income zip codes (where less than 17.5% of households earned less than $15,000). Household income explained 59% of the variation in CHF hospitalization rates at the zip code level.

Fosters true quality improvement

Physician practice style varies across areas, but does not explain variation in admission rates for chronic medical conditions after adjusting for community sociodemographic factors. Outpatient interventions such as the use of protocols for ambulatory management of low-severity patients, and improvement of access to outpatient care, would most likely decrease inpatient admissions for CHF. ⁹³ There is little evidence that lower rates of CHF admission would lead to worse patient outcomes. However, practice guidelines or utilization review intended to raise physicians' threshold for admission may not be effective in reducing hospitalizations for chronic medical conditions. ⁴⁹⁶

Prior use

This measure was originally developed by Billings and colleagues in conjunction with the Ambulatory Care Project of the United Hospital Fund of New York, ²⁷⁵ but a similar measure was developed contemporaneously by Weissman et al.. ²⁷⁶ This measure was subsequently adopted by the Institute of Medicine, ⁵⁸³ and has been widely used in a variety of studies of avoidable or preventable hospitalizations. At least 6 states (MA, NE, UT, VA, MI, NY) are reportedly using this set of measures "as guidance for policy and as evaluation and decision aids." ⁵⁸⁴ Internationally, CHF admissions are tracked by the United Kingdom as part of the UK National Health Service High Level Performance Indicators. ⁵⁹⁰ A related measure is included in the DEMPAQ ⁵⁹¹ measure set. A closely related indicator, nonelective admission for CHF, was recently recommended as a measure of access to care for elderly Medicare beneficiaries. ⁵⁸⁵

Precision

This indicator is very precise, with a raw area level rate of 521.0 per 100,000 and a standard deviation of 286.5. The systematic area level standard deviation is high, at 0.14%. The area level variation also accounts for a high percentage of total variation, at 0.04%. This means that relative to other indicators, a higher percentage of the variation occurs at the area level, rather than the discharge level. The signal ratio is very high, at 93.0%. This means that it is very likely that the observed differences in area performance represent true differences in area performance. The very high R-square reflects the large proportion of signal that can be extracted using multivariate techniques. Such techniques do not have substantial additional impact, primarily due to the already very high ratio.

Bias

Signal variance decreased by over 15% with risk adjustment, indicating that some of the true variation among providers reflects differences in patient demographic characteristics. The indicator performs fairly to well on multiple measures of minimum bias. The rank correlation is good at 0.858. Risk adjustment does appear to affect both the extremes of the distribution substantially, with only 54.5% of providers in the highest decile remaining after risk adjustment and 81.8% in the lowest decile. Further, 16.6% of providers move more than 2 deciles after risk adjustment. The absolute magnitude of the impact is moderate.

Construct validity

CHF is related to most other ACSC conditions.

Face validity

Diabetic ketoacidosis, hyperosmolarity (HHNS), and coma are life-threatening complications of diabetes mellitus, particularly type 1 or insulin dependent diabetes mellitus (IDDM). Diabetic emergencies arise when there is an excess of glucose or insulin. The balance of insulin and glucose is kept by proper administration of insulin, and may involve other activities such as home blood-glucose monitoring. It has been noted in an adolescent and young adult population that better adherence to treatment (actual insulin intake vs. prescribed intake) is associated with fewer admissions for ketoacidosis and other complications. ⁵⁹² Education programs for patients with diabetes have mixed results on reducing admissions for diabetic emergencies, though some have been shown to be effective. ⁵⁹³ It is important to note that intensive treatment (continuous insulin infusion pump, or multiple insulin injections daily) has beenassociated with more admissions for hypoglycemia. ⁵⁹⁴ Such intensive treatment has not been shown to have impact on admissions hyperglycemic events, but does reduce the incidence of long-term complications. Both hypoglycemic and hyperglycemic events are included in this indicator.

Minimum bias

Previously this indicator was defined with a hospital level denominator. Since some hospitals may be referral centers for diabetes, or may treat more difficult patients, some hospitals may have artificially high rates, though a quality problem is not present. This indicator has been redefined as an area-level measure. Some areas may have higher rates of diabetes, due to ethnic or age composition. It would be expected that these areas would have higher admission rates for diabetic emergencies. Other factors, such as illness,^595-597 may also predispose patients to be admitted for diabetic emergencies. However, it is unlikely that any one area would experience significantly higher rates of these factors.

Admissions for diabetic emergencies can occur in both patients with existing and treated diabetes, as well as patients with previously unknown diabetes. One New Zealand study of 196 patients admitted for DKA found that 20% of admissions were new onset diabetes. ⁵⁹⁸ Two separate US studies of a US Urban African-American population found that 25% and 17% patients admitted for DKA were reportedly new onset diabetes.^{595, 596}

Older age is associated with higher rates of underlying illness, more severe DKA, and better pre-hospitalization glycaemic control. This indicates that older patients may have fewer compliance issues and more complex cases. ⁵⁹⁷

Construct validity

Precipitating events leading to admission may include physiologic causes, as discussed above, or the cessation of treatment due to access to care or non-compliance issues. Evidence that such causes are or are not due to access to care contributes to the construct validity of this indicator. However, such evidence has not been strongly shown. Some studies outside the US, and a few inside the US have examined the precipitating events of admission for diabetic emergencies. These studies often rely on self-report, which may be a biased measurement in and of itself. Of patients with previously known and treated diabetes, over 60% had made an error in insulin administration or had omitted insulin. Few of these patients also had underlying illness. Further, 25% of the original patients were readmitted within the 18-month study period. This study has no indication whether or not these errors were due to non-compliance, poor education, or access to care problems. ⁵⁹⁸ A Scottish study of young adult patients found that 42% of DKA admissions were due to lack of adherence to insulin treatment. ⁵⁹⁷

In a potentially underserved population of Urban African-Americans, 2/3 of admissions were due to cessation of insulin therapy. Half of the patients stopping insulin treatment reported financial or other difficulties in obtaining insulin, while 21% reported inadequate understanding in adjusting dosages with food intake, and 14% were unsure about insulin management on sick days. Fourteen percent were clearly non-compliant. Most patients reported having been educated in diabetes care. ⁵⁹⁵ In a related study at a later date, 49% of patients with DKA, and 42% of patients with HHNS stopped or inadequately administered insulin prior the diabetic emergency. ⁵⁹⁶

Access to care in relation to admissions has been explicitly studied and reported. Weissman ²⁷⁶ found that uninsured patients had a higher risk of admission for DKA and coma than privately insured patients (adjusted O.R. 2.18 - 2.77). Bindman ²⁸⁴ reported that an area's self-rated access to care report explained 46% of the variance in admissions for diabetes, though the analysis was not restricted to diabetic emergencies.

Several studies, including Billings ²⁷⁵ and Pappas, ²⁸¹ showed that residents of low-income communities have a higher risk of "ambulatory care sensitive" admissions, including short-term diabetic complications, than residents of high-income communities. Of course, this is only indirect evidence of validity, because low income and high income communities may differ for many reasons other than access to care. In addition, these studies aggregated ambulatory-care sensitive admission rates across multiple conditions, so they do not clearly support the validity of component measures, such as admission rates for short-term diabetic complications. Two studies of ACSC indicators reported validation work for diabetes independent of measure sets. Millman et al. ⁴⁹¹ reported that low-income zip codes had 4.1 times more diabetes hospitalizations per capita than high-income zip codes in 11 states in 1988. Billings et al. ⁴⁸⁹ found that low-income zip codes in New York City (where at least 60% of households earned less than $15,000 in 1988, based on adjusted 1980 Census data) had 6.3 times more diabetes hospitalizations per capita than high-income zip codes (where less than 17.5% of households earned less than $15,000). Household income explained 52% of the variation in short term diabetes complication hospitalization rates at the zip code level.

Fosters true quality improvement

We found no evidence regarding the gaming of this indicator. Since diabetic emergencies are potentially life-threatening, it is unlikely that hospitals would fail to admit patients requiring hospitalization. Since this indicator is an area-level indicator, diversion to a nearby hospital is a non-issue.

Prior use

Admission for diabetic emergencies was included in both Billings ⁴⁸⁹ and Weissman's ²⁷⁶ sets of avoidable hospitalization measures. The indicator was also identified as a promising measure of quality and access by the DEMPAQ (Developing and Evaluating Performance Measures for Ambulatory Care Quality) ⁵⁹¹ project, supported by the US Health Care Financing Administration, and the ACE (Access to Care for the Elderly) project, supported by the Physician Payment Review Commission. However, the denominator for these latter measures is limited to patients known to have diabetes, based on inpatient or outpatient claims diagnoses. This indicator, defined as a provider-level indicator, is currently an HCUP I indicator.

Precision

This indicator is moderately precise, with a raw area level rate of 36 per 100,000 population and a standard deviation of 24.6. The systematic area level standard deviation is moderate, at 0.01%. The area level variation also accounts for a moderate amount of total variation, at 0.002%. This means that relative to other indicators, a lower percentage of the variation occurs at the area level, rather than the discharge level. The signal ratio is also moderate, at 51.7%. This means that it is likely that some of the observed differences in area performance do not represent true differences in area performance. Multivariate techniques do not appear to improve the amount of extractable signal, as is reflected by the moderate R-square.

Bias

Signal variance does not change with risk adjustment. The indicator performs very well on the multiple measures of minimum bias. The rank correlation is very good at 0.995, and risk adjustment does not appear to change the composition of providers in the highest and lowest decile. The absolute magnitude of the impact is also minimal.

Construct validity

Diabetes short term complications rate is related to most other ACSC indicators.

Precision

This indicator is moderately precise, with a raw area level rate of 34.7 per 100,000 population and a standard deviation of 28.1. The systematic area level standard deviation is moderate, at 0.01%. The area level variation also accounts for a moderate amount of total variation, at 0.007%. This means that relative to other indicators, a lower percentage of the variation occurs at the area level, rather than the discharge level. The signal ratio is high, at 72.6%. This means that it is likely that the observed differences in area performance represent true differences in area performance, though some represents noise. Multivariate techniques do not appear to improve the amount of extractable signal, as reflected by the high R-square.

Bias

Signal variance does not change with risk adjustment. The indicator performs well on the multiple measures of minimum bias. The rank correlation is very good at 0.976. Risk adjustment changes the composition of providers in the highest and lowest decile moderately. The absolute magnitude of the impact is also moderate.

Construct validity

Diabetes short term complications rate is related to most other ACSC indicators.

Face validity

Over 10.5 million people in the U.S. have been diagnosed with diabetes mellitus, with over 90% of those having type 2 diabetes (NIDDM). ⁵⁹⁹ Long term complications occur in the majority of diabetic patients to some degree, and include retinopathy (mild to proliferative), neuropathy, nephropathy, and microvascular disorders. ⁵⁹⁹

Several observational studies have linked improved glycemic control to substantially lower risks of developing complications (retinopathy, neuropathy and nephropathy) in both Type 1 and Type 2 diabetes. ⁶⁰⁰ One study found that reducing glycosylated hemoglobin levels (a measure of glycemic control) has been estimated to produce over a two percentage point decrease in lifetime risk of blindness, ⁶⁰¹ though the reduction is less for elderly patients. Another study of Type 2 diabetes reported a decrease of 60-100% in the incidence of retinopathy and macular edema, a 24%-50% decrease in gross protienuria, and a 16%-100% decrease in lower-extremity amputation, depending on the age of onset and insulin treatment. ⁶⁰² Other studies have confirmed the benefit of tighter glycemic control on complications of NIDDM.^{603, 604} It has been recommended that near-normal glycemic control be maintained in both NIDDM and IDDM. ⁶⁰⁵

One mechanism of improving glycemic control is intensive therapy, which generally includes the use of a continuous insulin pump or multiple injections of insulin daily. The largest randomized control trial of intensive therapy for Type 1 or IDDM diabetes was The Diabetes Control and Complications Trial. Intensive therapy slowed the progression and decreased the development of retinopathy by 54% and 47% respectively. Also reduced were the incidences of microalbuminuria and clinical neuropathy (39% and 69%). ⁶⁰⁶ Fours years post trial, the differences between the intensive therapy group and the control group narrowed, but remained significant. ⁶⁰⁷

Given that appropriate adherence to therapy, and consistent monitoring of glycemic control, help to prevent complications, high-quality outpatient care should lower long-term complication rates. However, adherence to guidelines aimed at reducing complications (including eye and foot examinations, and diabetic education) has been described as modest^{608, 609} with only 1/3 of patients receiving all essential services. ⁶¹⁰

Precision

Diabetes affects a large number of people, as do diabetic complications. Hospitalizations for amputations and other diabetic complications are not rare, ⁶¹¹ suggesting that reasonably precise estimates can be obtained. However, few studies have documented hospitalization rates for diabetic complications and the extent to which they vary across areas.

Minimum bias

It is possible that some sociodemographic characteristics of the population may lead to higher rates of long-term diabetic complications. Rates of diabetes are higher in Black, Hispanic and especially Native American populations than in other ethnic groups. Hyperglycemia appears to be particularly frequent among Hispanic and Native American individuals. ⁵⁹⁹ The duration of diabetes is positively associated with the development of complications. Since new-onset diabetes occurs more often in an elderly population, areas with older populations may have shorter durations of diabetes, and fewer long term complications. Though few studies have examined the validity of hospital diagnoses related to apparent long-term diabetic complications, one study found that administrative databases (including outpatient and lab data) had "disappointing" PPVs for long term complications (PPVs ranged from less than 50% to 88% for a three year period). ⁶¹² Whether such population differences and biases lead to substantial differences in diabetic complication rates across geographic areas is unclear.

Construct validity

As noted above, substantial evidence exists that compliance with treatment guidelines to prevent long-term complications of diabetes is low, and that long-term diabetic complications are common. However, the importance of problems in the quality of outpatient diabetes care in explaining variations in diabetic complication rates is less well understood. Compliance of physicians and patients is essential to achieve good outcomes, and it seems likely that problems with both access to and quality of care as well as patient compliance may contribute to the occurrence of complications.

Fosters true quality improvement

Little evidence exists on the impact of this quality improvement measure on the delivery of outpatient care for diabetes. Because the optimal hospitalization rate for this condition has not been defined, providers may decrease their rates by failing to hospitalize patients who would truly benefit from inpatient care. Although this concern cannot be dismissed, there is no published evidence of worse health outcomes in association with reduced hospitalization rates for long-term complications of diabetes. Such an effect seems implausible, given that only the most serious complications of diabetes are treated on an inpatient basis.

Prior use

This indicator, defined as a hospital-level indicator, is a current HCUP I indicator.

Precision

This indicator is moderately precise, with a raw area level rate of 80.8 per 100,000 population and a standard deviation of 58.1. The systematic area level standard deviation is moderate, at 0.03%. The area level variation also accounts for a moderate amount of total variation, at 0.009%. This means that relative to other indicators, a lower percentage of the variation occurs at the area level, rather than the discharge level. The signal ratio is high, at 75.6%. This means that it is likely that the observed differences in area performance represent true differences in area performance, though some is due to random noise. The high R-square reflects the high amount of signal that can be extracted using multivariate methods, though this amount is less than for other indicators. Multivariate techniques do not appear to improve the amount of extractable signal, as is reflected by the moderate R-square.

Bias

The signal variance does not change with age-sex risk adjustment. The indicator performs well on the multiple measures of minimum bias. The rank correlation is good at 0.926. Risk adjustment does not appear to impact the lowest or highest decile disproportionately. The absolute magnitude of the impact is moderate, and 8.3% of areas change more than two relative deciles with risk adjustment.

Construct validity

Long term diabetes is related to the other ACSC conditions.

Face validity

Hypertension is a chronic condition that is often controllable in an outpatient setting with appropriate use of drug therapy. We found little literature on hypertension admission as an ambulatory care sensitive condition indicator. Hypertension was originally included in both John Billings' ⁴⁸⁹ set of indicators developed for the United Hospital Fund of New York, and in Weissman's ²⁷⁶ set of indicators. These sets were developed by physician panels. Evidence on sets of ambulatory care sensitive condition indicators are summarized at the beginning of this section, and should be referred to for this indicator.

Precision

Although hypertension is a common condition, hospitalizations for complications of hypertension are relatively uncommon. One study noted that hypertension accounted for only 0.5% of total admissions for ACSCs. ²⁸²

Minimum bias

We found very little evidence on potential biases in this indicator. It is possible that the age structure of the population may affect admission rates for this condition. Weissman et al. reported a reduction of 100% in relative risk for Medicaid patients when adjusting for age and sex. ²⁷⁶ Though it seems plausible that differences in socioeconomic status and comorbid conditions such as obesity would affect population rates of hypertension and its complications, we found no evidence on the effects of comorbidities or other risk factors that may vary systematically by area on admission rates for hypertension complications in the area.

Construct validity

Two studies of ACSC conditions reported the results for hypertension independently. Bindman et al. found that an area's self rated access to care explained 22% of admissions for hypertension. ²⁸⁴ Weissman et al. found that uninsured patients had a relative risk of admission for hypertension of 2.38 in Massachusetts after adjustment for age and sex, while Maryland had a corresponding relative risk of 1.93. Medicaid patients also had somewhat elevated risks (adj. RR = 1.56, 1.74). ²⁷⁶ Millman et al. ⁴⁹¹ reported that low-income zip codes had 7.6 times more hypertension hospitalizations per capita than high-income zip codes in the same 11 states in 1988.

Fosters true quality improvement

Little evidence exists on the impact of this quality improvement measure on the delivery of outpatient care for hypertension. Because the optimal hospitalization rate for this condition has not been defined, providers may decrease their rates by failing to hospitalize patients who would truly benefit from inpatient care. Although this concern cannot be dismissed, there is no published evidence of worse health outcomes in association with reduced hospitalization rates for hypertension. Such an effect seems implausible, given that only the most serious episodes of accelerated or malignant hypertension are treated on an inpatient basis.

Prior use

This measure was originally developed by Billings and colleagues in conjunction with the Ambulatory Care Project of the United Hospital Fund of New York, ²⁷⁵ and was subsequently adopted by the Institute of Medicine. ⁵⁸³ It has been widely used in a variety of studies of avoidable or preventable hospitalizations. At least 6 states (MA, NE, UT, VA, MI, NY) are reportedly using this set of measures "as guidance for policy and as evaluation and decision aids." ⁵⁸⁴ This indicator was also included in Weissman's set of avoidable hospitalizations. ²⁷⁶

Precision

This indicator is moderately precise, with a raw area level rate of 37.1 per 100,000 population and a substantial standard deviation of 32.2. The systematic area level standard deviation is moderate, at 0.01%. The area level variation accounts for only a moderate percentage of total variation, at 0.006%. This means that relative to other indicators, a lower percentage of the total variation occurs at the area level, rather than the discharge level. The signal ratio is moderate, at 69.9%. This means that it is likely that some of the observed differences in area performance do not represent true differences in area performance. The high R-square denotes the high amount of signal that can be extracted using multivariate methods, though this is less than for other indicators. Multivariate methods do not appear to have substantial additional impact.

Bias

Signal variance does not change with risk adjustment. The indicator performs well on multiple measures of minimum bias. The rank correlation is very good at 0.963. Risk adjustment appears to affect the highest decile disproportionately to the lowest decile, as 72.7% of areas in the highest decile and 100% of the lowest decile remain after risk adjustment. Few providers change more than 2 deciles in relative performance with risk adjustment. The absolute magnitude of the impact is minimal.

Construct validity

Hypertension is related to the other ACSC conditions.

Face validity

Lower-extremity amputation is a common complication of diabetes, affecting up to 15% of all diabetics in their lifetimes. ⁶¹³ In the United States, diabetes is the leading cause of nontraumatic amputations (approximately 57,000 per year). ⁶¹⁴ While the full etiology of factors leading to amputation are unknown, it is believed that a combination of factors contribute to the high rate of amputation in the diabetic population. Neuropathy and the subsequent loss of sensation may lead to minor trauma to the feet. These lesions, including foot ulcers, may fail to heal due to poor circulation and other factors. Resulting infections may lead to gangrene.^{613, 615} Each of these singular causes can be prevented to a certain extent, leading to the prevention of lower extremity amputation. Possible interventions include foot clinics, wearing proper foot ware, and proper care of feet and foot ulcers. ⁶¹⁵ Therefore, the American Diabetes Association (ADA) recommends that diabetics be educated in proper foot care, and that diabetics undergo a foot exam at least once a year. In addition, the Diabetes Control and Complications Trial (DCCT) ⁶⁰⁶ found that some of these factors, including neuropathy and microvascular disease, are preventable by maintaining blood glucose levels near normal using intensive insulin therapy. The DCCT has led to further recommendations that blood-glucose levels should be closely monitored in all diabetics (2-4 measurements per year), and glycosylated hemoglobin levels should be maintained below 8%.

There is substantial evidence that the ADA recommendations are not closely followed. One study of a California HMO noted that over half of patients did not have documented glycosylated hemoglobin levels (a measure of blood glucose control). In addition almost 40% of those patients with documented levels had at least one high level noted (over 10%), denoting overall poor glycemic control. Almost all patients had no documented foot exams, though these may have occurred without documentation. ⁶¹⁶ In the median state in 1998-99, only 71% of Medicare beneficiaries had a glycosylated hemoglobin test within the previous year, while only 69% and 57% had an eye exam and a lipid profile, respectively, within the previous 2 years. ⁶¹⁷ Several older studies, involving both Medicare^{618, 619} and Medicaid ⁶²⁰ recipients, showed similar deficiencies in processes of care that may help prevent amputations. In addition, the Medical Outcomes Study and the National Health and Nutrition Examination Survey ⁶¹⁸ both showed that over 50% of diabetics are less than optimally controlled, and as many as 18.8% (in staff-model HMOs) to 32.4% (in fee-for-service care) may be at extremely high risk of complications, with a glycosylated hemoglobin of 12% or more. ⁶²¹

Precision

Within the diabetic community, the incidence of lower extremity amputation has been reported as 375 per 100,000 person years for NIDDM, and 388 per 100,000 person years for IDDM. The twenty-five year cumulative risk for lower extremity amputation was 11%. ⁶²¹

Although we located no studies discussing small-area variation of lower extremity amputation, worldwide numbers greatly vary from 2.8 per 100,000 population (Madrid, Spain) to 43.9 in the Navajo population in the United States. The other US study site, Montgomery, AL had an age adjusted incidence of 19.2 per 100,000. ⁶²²

Minimum bias

Several sociodemographic variables are associated with the risk of lower-extremity amputation in diabetics, including age, duration of diabetes, and sex.^{613, 623} Males have been found repeatedly to have higher risks of amputation (2.8-6.5 fold higher rates). ⁶¹³ Age and sex distribution may vary systematically by area.

Race appears to be an important factor that may influence area rates. A study of hospital discharges in 1991 in California noted that African Americans had just under twice the rate of amputation as Whites (95.25 vs. 55.98 per 10,000 persons with diabetes, RR=1.72). Hispanics had a rate similar to that among Whites (44.43 per 10,000 persons with diabetes). ⁶²⁴ While minorities may have up to twice the amputation rates as whites, ⁶¹³ it is unknown whether this association is due to differences in access to quality care, compliance, or biological risk factors. Another study of risk factors leading to amputation noted that in an insured HMO population, African-Americans did not have greater risk of amputation, ⁶²³ suggesting that the observed differences by race are due to access to care, and should not be adjusted for. However, rates of diabetes are uniformly higher in Black, Hispanic and especially Native American populations than in other ethnic groups. Hyperglycemia appears to be particularly frequent among Hispanic and Native American individuals. ⁵⁹⁹ Therefore, adjusting or stratifying by race may be advisable when this indicator is defined using all adults, rather than all adult diabetics, as the denominator population.

Two controlled studies have identified clinical risk factors for amputation among diabetics in an HMO ⁶²³ and in a VA medical center, ⁶²⁵ both settings with relatively good access to care. Selby and Zhang found that the level of glucose control, duration of diabetes, and baseline systolic blood pressure were major clinical predictors of amputation. Other diabetic complications, such as microvascular complications and history of stroke, were also predictive. ⁶²³ Reiber et al. controlled for sociodemographic factors, and found that sensory perception, circulation indices, and nutritional factors were associated with amputation risk. Patients with the most severe disease had a four-fold risk of amputation, compared to patients with less severe disease. ⁶²⁵ These studies suggest that some areas may have higher amputation rates than others, partially because of unmodifiable patient characteristics (e.g., duration of diabetes). However, many of the clinical factors associated with amputation are potentially modifiable in the long term, if excellent control of hyperglycemia and hypertension are maintained. Therefore, the magnitude of potential bias due to confounding depends somewhat on whether one takes a short-term or long-term perspective.

Construct validity

Several studies of intervention programs have noted a decrease in amputation risk. A 1989 prospective randomized study of a 1 hour foot care education program for high risk patients (patients with foot ulcers or previous amputation), noted a 3-fold greater risk of amputation 2 years after intervention in control patients. The education program informed patients of proper foot care and was supplemental to normal teaching regarding diet, exercise, weight and medication. ⁶²⁶ A more recent study noted a 1 year post-intervention decrease of 79% in amputations in a low-income African American population. The intervention varied by risk, with low risk patients receiving foot care education, and assistance in finding proper fitting footwear. High risk patients were provided with custom-molded orthoses and prescription footwear. Additional foot care was provided for patients with foot injuries. ⁶²⁷

One study examining the literature noted that provider and patient education may lead to a 72% decrease in amputations, multidisciplinary clinic care, a 47% decrease, and insurance coverage for therapeutic shoes a 53.5% decrease. They calculated the potential economic benefits for the first year to be over $1.1 million, $750,000, and $850,000 for educational interventions, multidisciplinary clinics and insurance coverage for footwear respectively. However, most of the benefit would occur in individuals 70 years or older. ⁶²⁸

One observational study of the risk factors of lower-extremity amputation found that patients who receive no outpatient diabetes education have a three-fold higher risk of amputation than those receiving care. ⁶²⁵ Although there is no clear evidence that areas with higher amputation rates provide worse care to diabetic patients, the evidence certainly suggests that high-quality care can substantially decrease amputation rates among diabetics.

Fosters true quality improvement

We located no evidence regarding the ability of this indicator to foster true improvement. It is unlikely, given the severity of conditions requiring lower-extremity amputations, that patients requiring amputation would be denied care.

Prior use

This indicator is not widely used. Healthy People 2010 ⁵ has set a goal of reducing the number of lower extremity amputations from 4.1 per 1,000 persons with diabetes (in 1997) to 1.8 per 1,000 persons with diabetes. In addition this indicator is included in the DEMPAQ measure set for outpatient care.

Precision

This indicator is moderately precise, with a raw area level rate of 30.5 per 100,000 population and a substantial standard deviation of 42.7. The systematic area level standard deviation is moderate, at 0.04%. The area level variation accounts for only a moderate percentage of total variation, at 0.001%. This means that relative to other indicators, a lower percentage of the total variation occurs at the area level, rather than the discharge level. The signal ratio is moderate, at 68.5%. This means that it is likely that some of the observed differences in area performance do not represent true differences in area performance, though some also reflects random noise. The high R-square denotes the high amount of signal that can be extracted using multivariate methods, though this is less than for other indicators. Multivariate methods do not appear to have substantial additional impact.

Bias

Signal variance does not change with risk adjustment. The indicator performs well on multiple measures of minimum bias. The rank correlation is good at 0.919. Risk adjustment appears to affect the highest and lowest decile somewhat, as 59.1% of areas in the highest decile and 90.9% of the lowest decile remain after risk adjustment. The absolute magnitude of the impact is moderate.

Construct validity

Lower extremity amputation is slightly related to the other ACSC conditions, though it appears to be somewhat independent, as it loads on factor 2 more highly.

Face validity

Infants may be low birth weight due to inadequate interuterine growth or premature birth. Once premature labor has commenced, it is often quite difficult to stave the progression for any significant amount of time. However, risk factors for low birth weight may be addressed with adequate prenatal care and education. Risk factors include many sociodemographic and behavioral characteristics, such as low income and tobacco use during pregnancy. Prenatal education and care programs have been established to help reduce low birth weight and other complications in high risk populations. Addressable risks include: maternal undernutrition, genital tract infections, excessive physical exertion or stress, psychological stress and adverse health habits such as nicotine, alcohol or illicit drug exposure or poor prenatal care. ⁶²⁹ Nonetheless, evidence of the effectiveness of these programs has been equivocal. It is unclear whether increasing prenatal care or education actually reduces low birth weights. Healthy People 2010 has set a goal to reduce the percentage of low birth weight infants to .9%. ⁵

Precision

Although low birth-weight births account for only a small fraction of total births, the large number of births suggest that this indicator should be precisely measurable for most areas.

Minimum bias

Of the risk factors for low birth weight, very few are related directly to patient care. It is unclear how many of the risk factors are in actuality indirect measures to problems accessing care. Socioeconomic measures such as parental education and income have been shown to be negatively associated with rates of low birth weight infants.^{630, 631} Demographic factors such as age and race also appear important, and may be correlated with socioeconomic factors. Very young mothers (under 17 years) and older mothers (over 35 years) are at a higher risk of having low-birth weight infants.^{630, 631} Other factors such as tobacco use, primiparity, complications of pregnancy or labor and delivery, and marital status have also been cited as risk factors for low birth weight. ⁶³⁰

Black race has been repeatedly shown to be a risk factor for low birth weight. Many studies have attempted to control for the many confounding risk factors discussed above. One study of all California singleton births in 1992 found that after risk adjustment having a black mother remained a significant risk factor (Adj. OR = 1.6). ⁶³⁰ In an attempt to delineate whether maternal birthplace affected low birth weight among black and other minority groups. David et al. ⁶³² examined the relative risks of black mothers in Illinois born in the US, West Africa, or white mothers born in the US. They found that African born women had a 50% increase in relative risk as compared to white mothers, and US born black mothers had a 100% increase, when matching cases with respect to age, marital status, education and spouses education, prenatal care, parity and previous prenatal loss. Another study that examined California births in 1992 found no difference between foreign born and US born black women when adjusting for maternal and infant characteristics. ⁶³³ As this study adjusted for more characteristics than the David et al. study, the difference found in the David et al. study may be due to such bias. This study did find a difference between Latina US born and foreign born women, with US born women having higher rates of low-birth weight infants, even after adjustment for maternal and infant characteristics.

Low births weight also varies systematically by metropolitan versus non-metropolitan areas. An analysis of 11 million births in all 50 states from 1985-1987 found that mothers residing in non-metropolitan counties more likely to have low birthweight infants than those residing in metropolitan counties, before risk adjustment. However, after adjusting for maternal race, age less than 18, age over 35, nulliparity, parity greater than 4, single marital status, completion of high school and to a limited extent late prenatal care (this was considered an outcome as well), there were no longer statistical differences between metropolitan and non-metropolitan areas. This suggests that this indicator at an area-level could be potentially biased. ⁶³⁴

The interrelationship between all these risk factors and prenatal care and complications are very complex and have not been studied in a large sample. One study of women in Baltimore noted that factors of potential social stress, such as crime rate or unemployment rate, may interact with other risk factors, creating a complicated web of risk factors. ⁶³¹ Indeed, the picture of which factors lead to low birth weight is unclear because of these complex relationships.

Little evidence exists on the extent to which each of these factors contributes to differences in the rate of low birthweight births across geographic areas.

Construct validity

A number of studies have addressed the impact of prenatal care, or level of prenatal care among low birth weight babies. One study examined birth records in California to establish the relative risk of having a low birth weight baby as a function of use of prenatal care. Those with inadequate care (as calculated by Kotelchuck's Adequacy of Prenatal Care Use Index) had a 3.68 adjusted odds ratio of having a low birth weight infant, adjusting for maternal and infant characteristics as well as insurance status. It is important to note that those with more than adequate care had an adjusted odds ration of 6.78, with the same adjustment. These are likely to be high-risk patients that were followed closely and suggests that some of relevant risk adjustment factors were not included in this model. ⁶³⁰

One randomized control trial examined the effect of prenatal care on reducing low birth weight rates in low risk women (women with past high-risk obstetrical complications, current high risk conditions, or significant comorbidities). Reducing prenatal care from 14 visits to 9 visits did not affect the rate of low birth weight infants. However, there is no indication of the socioeconomic profile of the participants and their non-clinical risk factors. ⁶³⁵

Finally, one observational study, difference in the use of prenatal care accounted for less than 15% of the differences between low birth weight in black and white mothers enrolled in Kaiser-Permanente. However, increasing level of prenatal care was associated with lower rates of low birth weight, particularly in the black patient population. ⁶³⁶

One review of studies evaluating programs to reduce low birth weight notes that the studies have often been poorly designed, and this lack of rigor may account for some of the equivocal findings. The authors argue that prevention programs aimed at one specific risk factor in a population shown to have high rates of that factor do reduce low-birth rates, while comprehensive programs in potentially high risk populations do not reduce low-birth weight. They argue that this is potential evidence that prevention programs are simply misapplied and have poor designs. ⁶²⁹ Thus, while specific studies have demonstrated an impact of particular interventions, especially in high-risk populations, evidence on the impact of better prenatal care on low birthweight rates for area populations is less well developed.

Fosters true quality improvement

It seems unlikely that use of this indicator could lead to apparent reductions in the rate of low birthweight births that did not represent true reductions.

Prior use

Low birth weight has been used as a quality indicator on a limited basis, though interest in preventing low birth weight has been demonstrated through the literature and implementations of prevention programs. Low birth weight is a indicator in the HEDIS measure set for insurance groups, and is used by United Health Care and the University Hospital Consortium. The previous version of the HCUP I indicator set included both low birth weight and very low birth weight indicators. Healthy People 2010 has set a goal to reduce the percentage of low birth weight infants to .9%. ⁵

Precision

This indicator is precise, with a raw area level rate of 3.9% and a standard deviation of 2.3%. The systematic area level standard deviation is very high, at 1.18%. The area level variation accounts for a very high percentage of total variation, at 0.27%. This means that relative to other indicators, a higher percentage of the variation occurs at the area level, rather than the discharge level. The signal ratio is moderate, at 67.1%. This means that it is likely that some of the observed differences in area performance do not represent true differences in area performance. However, the high R-square reflects the high amount of signal that can be extracted using multivariate techniques, though this is still lower than for other indicators.

Bias

No analyses were conducted, since all newborns are the same age, and linkage to maternal records is not available.

Construct validity

Low birth weight inversely related to the other ASCS indicators. It is positively related to perforated appendix rate.

Face validity

Asthma is the most common chronic disease in childhood and is one of the most frequent admitting diagnoses in children's hospitals.^{637, 638} In the United States, asthma affects an estimated 4.8 million children and adolescents, and in 1993 it was the cause of 198,000 admissions and 342 deaths in persons aged 24 years and younger. ⁶³⁷ There are effective ambulatory treatments for asthma as well as guidelines published by the National Heart, Lung, and Blood Institute, which are endorsed by the American Academy of Pediatrics. ⁵²⁰ These guidelines emphasize the importance of patients' access to care, appropriate diagnosis of asthma, establishment of a physician-patient relationship, timely management of asthma symptoms with appropriate medications, appropriate prophylactic and maintenance therapy, and adequate follow-up care. Healthy People 2010 has set a goal toreduce the admission rate for asthma to 25 per 10,000 population for children under 5 years, and 7.7 per 10,000 population for people aged 5-65 years. ⁵

Precision

Because asthma is one of the most common reasons for pediatric hospitalization, with an average rate of 28.0 per 10,000 children less than 15 years of age, ⁶³⁷ relatively precise estimates of asthma admission across areas or hospitals can be obtained. Admission rates for asthma tend to be higher during peak times of viral respiratory infections (Winter) and allergy seasons (Spring and Fall), so care must be taken to ensure a consistent time period for measurement. There is wide variation across areas in admission rates for asthma, so random variation from year to year may be important for less populated areas

Minimum bias

Some admissions with asthma are unavoidable and appropriate. For example, some children have especially severe disease due to genetic factors, associated cystic fibrosis or bronchopulmonary dysplasia, or increased exposure to environmental triggers.^{275, 560, 637, 639-641} Indoor allergens such as cockroaches and dust mites may be more common in lower-income areas, and are probably associated with increased frequency and severity of asthma symptoms ⁵²³ Tobacco smoke is the most important indoor irritant and is a major precipitant of asthma symptoms in both children and adults.^{520, 525-527, 529} Exposure to maternal tobacco smoke is a risk factor for the development of asthma in infancy ⁶⁴² and childhood,^643-649 although not for persistence of childhood asthma into adulthood. ⁶⁵⁰

Outdoor air pollution, especially respirable particulates, may also play a role.^{520, 528-532} In addition, ozone and SO₂ have been associated with increased emergency department visits and hospitalizations rates.^531-538 Increasing air pollution has been specifically correlated with higher admission rates in London (Ozone, NO₂, SO₂, and black smoke), ⁵³⁹ and Seattle (ambient air pollution) ⁵⁴⁰ .

Race represents one of the most complex potentially biasing factors for this indicator. Black patients have consistently been shown to have higher asthma admission rates,^544-546 even when stratifying for income and age. ⁵⁴⁷ One study examining differences in asthma health care utilization noted that African Americans made fewer asthma-related primary care and specialist visits than Caucasian patients (47.6% vs. 70.2% and 27% vs. 38.8%). There were no differences in hospitalization rates by race, but African-American patients had lower household incomes and made more emergency department visits (proxy for either access to care or asthma severity). ⁵⁴⁸ Similarly, Hispanics have been shown to have higher admission rates than non-Hispanic whites (or areas with higher percentages of Hispanics have been shown to have higher admission rates), although none of these studies controls for SES.^{545, 549, 550} To the extent that true differences in disease prevalence or severity are responsible for racial variation in hospitalization rates, race should be adjusted for in comparing asthma hospitalization rates across areas. On the other hand, to the extent that minority patients have less access to care or poorer quality of outpatient care, race should not be adjusted for.

Construct validity

Little evidence has been reported conclusively attaching poor quality of care to higher area admission rates. However, numerous studies have shown that asthma hospitalization rates are associated with socioeconomic factors, including median household income (at the area level) and lack of insurance (at the individual level). A study of asthma hospitalization rates in California in 1993 (ages 0-64) found that areas with median household incomes under $35,000 had hospitalization rates that were 1.5 times higher than areas with higher median incomes. ⁵⁴⁷ In Boston, in 1992, age and gender standardized hospitalization rates (all ages) were correlated with percentage poverty in an area (r=0.68), percentage holding a bachelor's degree (r=-0.61), and income (r=-0.51). ⁵⁵⁰ Within New York City in 1994, asthma hospitalization rates were negatively correlated with a zip code area's median household income (r=-0.67), and positively correlated with the percentage of minorities in the population (r=0.82). ⁵⁴⁹ These findings confirm an earlier study by Billings et al., ⁴⁸⁹ who reported 6.4-fold variation in asthma hospitalization rates at the zip code level in New York City in 1988, with 70% of this variation explainable by the percentage of households with annual income below $15,000. Millman et al. ⁴⁹¹ reported that low-income zip codes had 5.8 times more asthma hospitalizations per capita than high-income zip codes in 11 states in 1988. Using New York State data, Lin et al showed that hospitalization rates were higher in areas with higher poverty, unemployment, minority populations, and lower education levels. ⁵⁴⁵ Even in England, 45% of the variation in asthma hospitalization rates across 90 family health services authorities in 1990-95 was attributable to socioeconomic factors, plus the availability of secondary care. ⁵⁵¹ To our knowledge, only one study has reported partial correlations; ⁵⁵² it found that that in New York City, the percentage of African-American residents was the strongest predictor, and median household income was the next strongest predictor, of asthma hospitalization rates.

The observation that asthma admission rates are higher in areas with low SES has led some researchers to hypothesize that lack of access to care, or poor quality outpatient care, may lead to higher admission rates. Bindman et al. ²⁸⁴ showed that asthma hospital?ization rates across 41 sampled areas in California were significantly correlated (r=0.47) with self-rated access to needed medical care, according to community telephone surveys. Although analyses of the National Health and Nutrition Examination Survey found that Medicaid enrollment and Spanish language preference were associated with inadequate asthma therapy, these deficiencies in care were not directly linked to hospitalizations. ⁵⁵⁴ Studies from other settings have shown that African-American asthmatics tend to have fewer scheduled primary care visits, and more hospitalizations and emergency room visits, than White asthmatics.^{555, 556} African-Americans' use of asthma medications may also be less consistent with current practice guidelines. ⁵⁵⁷

Few studies have directly linked high-quality processes of outpatient care with lower hospitalization rates at either the area or the individual level. An in-depth study of asthma treatment practices in New Haven, Boston, and Rochester found that the community with the highest asthma hospitalization rate (Boston) also had lower use of inhaled antiinflammatory agents and oral steroids. The threshold for admission also appeared to be lower in Boston, as fewer of the admitted patients were hypoxemic, relative to the other cities. ⁵⁶⁰ One case control study from a large health maintenance organization established that not having a written asthma management plan was a strong risk factor for asthma hospitalization (after adjusting for severity of asthma), but the use of anti-inflammatory medications was not. ⁵⁶¹ With patient and parent education, good medical therapy, and outreach programs, adverse outcomes can be reduced considerably.^{561, 651}

Fosters true quality improvement

Because the optimal hospitalization rate for this condition has not been defined, providers may decrease their rates by failing to hospitalize patients who would truly benefit from inpatient care, or by hospitalizing marginally appropriate patients with other conditions (to inflate the denominator). Although these concerns cannot be dismissed, there is no published evidence of worse health outcomes in association with reduced hospitalization rates for asthma. Indeed, given studies showing high rates of inappropriate hospitalization and poor adherence to professional guidelines, a shift to outpatient care may be entirely appropriate. ⁶⁵² This is an area that should be further studied.

Prior use

This measure was originally developed by Billings and colleagues in conjunction with the Ambulatory Care Project of the United Hospital Fund of New York, ²⁷⁵ but a similar measure was developed contemporaneously by Weissman et al. ²⁷⁶ It was subsequently adopted by the Institute of Medicine, ⁵⁸³ and has been widely used in a variety of studies of avoidable or preventable hospitalizations. At least 6 states (MA, NE, UT, VA, MI, NY) are reportedly using this set of measures "as guidance for policy and as evaluation and decision aids. ²⁷⁶ The measure was developed to include all ages, but has since been adapted by Gadomski ²⁷⁷ and McConnochie ²⁸⁷ as a pediatric measure. South Carolina included asthma in the set of ambulatory care sensitive measures for pediatrics used by the South Carolina Department of Health Statistics. ²⁷⁸ Healthy People 2010 has set a goal to reduce the admission rate for asthma to 25 per 10,000 population for children under 5 years, and 7.7 per 10,000 population for people aged 5-65 years. ⁵

Precision

This indicator is precise, with a raw area level rate of 154.1 and a standard deviation of 143.9. The systematic area level standard deviation is high, at 0.11%. The area level variation also accounts for a high percentage of total variation, at 0.005%. This means that relative to other indicators, a higher percentage of the variation occurs at the area level, rather than the discharge level, though still lower than for other indicators. The signal ratio is high, at 85.1%. This means that it is likely that the observed differences in area performance reflect true differences in performance, though some reflects random noise. The high R-square reflects the high proportion of signal that can be extracted using multivariate methods, though lower than for other indicators. Multivariate techniques have little additional impact.

Bias

The signal variance does not change with risk adjustment. The indicator performs very well on multiple measures of minimum bias. Risk adjustment does not appear to affect the extremes of the distribution substantially. The rank correlation is very good at 0.994. No of areas move more than 2 deciles in relative performance. The absolute magnitude of the impact is minimal.

Construct validity

Pediatric asthma is related to most other ACSC indicators.

Face validity

Gastroenteritis is a common illness in childhood, resulting in nearly 200,000 hospitalizations annually (nearly 10% of all admissions of children under 5 years of age). ⁶⁵³ There are effective ambulatory treatments for gastroenteritis and clear guidelines published both by the Centers for Disease Control and the American Academy of Pediatrics. ⁶⁵³ These guidelines emphasize the importance of appropriate oral rehydration therapy for mild to moderate dehydration resulting from gastroenteritis, to avoid the need for hospitalization. Adherence to these guidelines is poor; only about 33% of children treated for gastroenteritis received glucose-electrolyte solutions recommended by the AAP. Most physicians instead recommend clear liquids, which have been shown to be ineffective in treating dehydration due to gastroenteritis,^654-656 and withholding solid food longer than the 24 hours recommended by the AAP. A physician panel agreed that timely and effective ambulatory care would reduce the risk of hospitalization for gastroenteritis. ²⁷⁵

Precision

Because gastroenteritis is one of the most common reasons for pediatric hospitalization, with small area rates of 200-400 per 100,000 children, relatively precise estimates of gastroenteritis admission across areas or hospitals can be obtained. Gastroenteritis is known to vary seasonally, with about half of hospitalizations occurring between February and April. ⁶⁵⁷ Thus the stability of the measure over time will be influenced by seasonal fluctuation in disease prevalence, so care must be taken to ensure a consistent time period for measurement. There is wide variation across areas in admission rates for gastroenteritis (14- to 18-fold differences), ⁶⁵⁷ so random variation in a particular year may be considerable for less populated areas and smaller hospitals.

Minimum bias

Some admissions with gastroenteritis are unavoidable and appropriate. For example, some children with gastroenteritis also suffer from a chronic disease, or from another infection such as gingivostomatitis or tonsillitis, that inhibits oral intake of liquids. ⁶⁵² These "mandatory admissions" may be difficult, if not impossible, to identify from HCUP data. However, most (73%) children admitted with gastroenteritis appear to have no underlying problems, and most (79%) are re-hydrated within 12 hours. One study suggests that complicated gastroenteritis admissions may be more common among children of low socioeconomic status. ⁶⁴⁰ If true, this finding suggests that clinical characteristics may explain some of the observed variation in admission rates for pediatric gastroenteritis.

Construct validity

No published studies have specifically addressed the construct validity of this indicator. Billings' original study from New York reported 1.87-fold variation in gastroenteritis hospitalization rates, with a coefficient of variation of 0.438 and 22% of variance explained by household income. ⁴⁸⁹ Millman et al. ⁴⁹¹ reported that low-income zip codes had 1.9 times more pediatric gastroenteritis hospitalizations per capita than high-income zip codes in the same 11 states in 1988.

Fosters true quality improvement

Because the optimal hospitalization rate for this condition has not been defined, providers may decrease their rates by failing to hospitalize patients who would truly benefit from inpatient care, or by hospitalizing marginally appropriate patients with other conditions (to inflate the denominator). Although these concerns cannot be dismissed, there is no published evidence of worse health outcomes in association with reduced hospitalization rates for gastroenteritis. Indeed, given studies showing high rates of inappropriate hospitalization and poor adherence to professional guidelines, a shift to outpatient care may be entirely appropriate. ⁶⁵² This is an area that should be further studied.

One evaluation of an intervention to improve access reported specifically on pediatric gastroenteritis. Kaestner et al. found no narrowing of the differences in "discretionary" infant (<2 year) hospitalization rates between low, middle, and high-income zip codes, during a period of substantial Medicaid eligibility expansion (1988-1992). ⁵⁰³ Disaggregation of gastroenteritis hospitalizations did not alter this finding.

Prior use

This measure was originally developed by Billings and colleagues in conjunction with the Ambulatory Care Project of the United Hospital Fund of New York. ²⁷⁵ It was subsequently adopted by the Institute of Medicine, ⁵⁸³ and has been widely used in a variety of studies of avoidable or preventable hospitalizations. At least 6 states (MA, NE, UT, VA, MI, NY) are reportedly using this set of measures "as guidance for policy and as evaluation and decision aids." ⁵⁸⁴ The measure was developed to include all ages, but has since been adapted by Gadomski ²⁷⁷ and McConnochie ²⁸⁷ as a pediatric measure. South Carolina included gastroenteritis in the set of ambulatory care sensitive measures for pediatrics used by the South Carolina Department of Health Statistics. ²⁷⁸

Precision

This indicator is precise, with a raw area level rate of 98.5 and a standard deviation of 10.1. The systematic area level standard deviation is high, at 0.05%. The area level variation accounts for a moderate percentage of total variation, at 0.03%. This means that relative to other indicators, a lower percentage of the variation occurs at the area level, rather than the discharge level. The signal ratio is high, at 77.8%. This means that it is likely that the observed differences in area performance represent true variation in performance, though some may also reflect random noise. The high R-square reflects the high proportion of signal that can be extracted using multivariate techniques, though multivariate techniques do not have substantial additional impact.

Bias

Signal variance does not change with risk adjustment. The rank correlation is very good at 0.995. Risk adjustment does not appear to impact extremes of the distribution substantially, as 90.9% of areas in the highest decile and 100% of the lowest decile remain after risk adjustment. No areas move more than two deciles in relative performance. The absolute impact is also minimal, with an average change in performance relative to the mean of 5.5%.

Construct validity

Pediatric gastroenteritis is related to most other ACSC indicators.

Face validity

Acute myocardial infarction (AMI) affects 1.5 million people each year and approximately one-third die in the acute phase of the heart attack. ⁶⁵⁸ Many clinical and observational studies have been conducted showing processes of care linked to survival improvements. These research findings have resulted in detailed practice guidelines covering all phases of AMI management. ⁴⁸⁶ Starting in 1992, the Health Care Financing Administration implemented a national initiative to gather data for quality improvement. The project, "the Cooperative Cardiovascular Project (CCP)," focuses on improving treatment of AMI patients.

Precision

The precision of AMI mortality rate estimates may be problematic for medium and small hospitals. About 13% of AMI patients in the California Hospital Outcomes Project died during hospitalization, or within 30 days of admission. ⁶⁵⁹ Since 19.5% of AMI patients were transferred from their initial hospital to another acute care facility, the percentage of deaths in an unlinked episode of care would have been somewhat less. In a study using the 1992 MedisGroups Comparative Database with 100 hospitals, mostly in Pennsylvania and the southern United States, the in-hospital mortality rate was 13.2% for AMI patients. In elderly Medicare AMI patients, 30-day mortality rates varied from 18% in Connecticut to 23% in Alabama. ⁶⁶⁰ Although these outcome rates are high, the number of AMI patients varies widely across hospitals, based on the size and risk profile of each hospital's catchment area.

Minimum bias

Starting in 1990, ICD-9-CM included a fifth digit for AMI codes to distinguish treatment during the "initial episode of care" from subsequent treatment related to the same AMI (within 8 weeks of the event). In studies comparing chart and administrative data since this time, the agreement in identification of new AMI cases has been shown to be at least 93%, and as high as 98%.^{659, 661} The California Hospital Outcomes Project found that "unlikely" AMI patients had significantly higher mortality than patients with definite or possible AMI. ⁶⁵⁹ However, there was no evidence of systematic bias across hospitals; high-mortality and low-mortality hospitals had similar proportions of "unlikely" AMI patients.

About 19.5% of AMI patients were transferred from the initial hospital to another acute care facility in the California Hospital Outcomes Project, and these transfer rates varied across hospitals. ⁶⁵⁹ In studies using unlinked data, hospitals transferring a large proportion of their AMI patients may have lower death rates than hospitals that do not regularly transfer patients. A related bias results from the fact that many deaths related to AMI occur after hospital discharge, but within 30-days. ⁶⁶² Thus, as described below (under "Fosters True Quality Improvement"), hospitals with long mean LOS may appear to have higher mortality rates than hospitals with shorter mean LOS. Investigating hospital LOS and transfer rates in conjunction with AMI mortality may help resolve these concerns.

Risk adjustment

Numerous studies have established the importance of risk adjustment for AMI patients. As a result, researchers have developed a number of risk adjustment models. Normand et al developed and validated two models, one of which was based on conditions likely to be present on admission and therefore applicable to comparisons of hospital-based care. ⁸⁴ The claims-based model included 25 comorbidities not related to treatment. Hypertension (18.3%), diabetes (13.8%), and pulmonary disease (11.2%) were the most frequent comorbidities in an AMI Medicare cohort of 164,427 patients. Examples of frequent comorbidities that were considered possibly related to hospital treatment, and therefore omitted from their model, included congestive heart failure (33.9%), chronic angina (27.4%), and arrhythmias (25.2%). The same team developed another model using the clinical predictors available from the Cooperative Cardiovascular Project. ⁸⁴ From these and numerous other studies, the most important predictors of short-term AMI mortality have been shown to include age, previous AMI, tachycardia, pulmonary edema and other signs of congestive heart failure, hypotension and cardiogenic shock, anterior wall and Q-wave infarction, cardiac arrest, and serum creatinine or urea nitrogen. Fewer studies have addressed whether adjusting for potential complications as well as comorbidities, or adjusting only for predictors available from administrative data, leads to bias in comparisons across hospitals.

Krumholz et al compared seven models including a newly developed 7-variable clinical/demographic risk adjustment model for 30-day mortality in AMI patients. ¹²³ The models based on clinical data demonstrated better discrimination and calibration than two models ⁶⁶⁰ based on ICD-9-CM codes (area under the receiver operating curve 0.74-0.78 versus 0.70-0.71, respectively). In addition, the clinical models classified hospital performance somewhat differently than the models based on administrative data. Such differences were further explored by Iezzoni and colleagues, who used several proprietary products to estimate risk-adjusted AMI mortality, and found 40-60% disagreement in identifying the 10 best and 10 worst hospitals in a nationwide sample.^{9, 204} Adding full clinical data to administrative data for risk-adjustment, Pine found that 73% of Cleveland hospitals' expected mortality rates changed by less than one standard deviation, and 100% changed by less than two. ¹⁹ In St. Louis, 95% of hospitals' expected mortality rates changed by less than 0.5 standard deviations, and 100% changed by less than one. These estimates were better than those for other major medical conditions, including pneumonia, stroke, and congestive heart failure. ²⁰⁵ In the California Hospital Outcomes Project, the addition of clinical risk factors to a reestimated model based on reabstracted ICD-9-CM codes had a minimal effect on the difference in risk-adjusted mortality between low-mortality and high-mortality hospitals, although individual hospitals were affected. ⁶⁵⁹ In summary, these studies found that the method of risk-adjustment does affect which specific hospitals are identified as mortality outliers, but that the correlations within pairs of risk-adjusted or expected mortality rates are generally high (e.g., 0>0.80) ¹² to 0.94 ²⁰⁵ , and higher for AMI than for other medical conditions.

When risk adjustment models include ICD-9-CM conditions that may represent consequences of poor care, then discrimination is exaggerated. ¹²³ Romano and Chan compared an administrative data set to a re-abstraction of diagnoses present at admission, with two versions of the All Patient Refined-Diagnosis-Related Groups (APR-DRG), Risk of Mortality (ROM) and Severity of Illness (SOI). ¹¹³ The authors showed empirically that APR-DRGs predicted 30-day mortality better when all diagnoses were included than when only diagnoses present at admission were included. Hospitals' expected mortality rates based on all reabstracted ICD-9-CM codes were moderately correlated (r=0.72-0.77) with expected mortality rates based only on diagnoses present at admission. However, 2 of the 3 hospitals classified as having higher than expected mortality, 8 of the 23 hospitals classified as having neither higher nor lower than expected mortality, and 0 of the 4 hospitals classified as having lower than expected mortality, switched categories when diagnoses not present at admission were excluded from risk-adjustment.

Construct validity

Numerous randomized controlled trials have conclusively demonstrated that early administration of aspirin and thrombolytic agents can reduce AMI mortality.^663-667 Similarly, early revascularization by percutaneous coronary angioplasty reduces mortality in high-risk patients.^668-670 Angiotensin converting enzyme inhibitors reduce mortality among post-infarction patients with impaired left ventricular function.^671-673 Therefore, there is clear evidence at the patient level that specific processes of care improve patient outcomes. Furthermore, numerous studies based on large regional or national samples have shown substantial practice variation in AMI patients, with underutilization of clearly beneficial therapies and overutilization of harmful treatments.^{95, 674, 675}

Over the last several years, substantial evidence for construct validity at the hospital level has emerged. In the first study of this type, Park et al. estimated the contribution of differences in severity of illness and quality of care to the classification of some hospitals as having unexpectedly high inpatient death rates (age and gender adjusted). ³⁸ Not unexpectedly, severity of illness (using chart data) accounted for some of the variation. However, a quality score derived from an explicit set of process measures did not explain differences between low-mortality and high-mortality hospitals. In fact, the relationship was in the opposite direction from the authors' expectation under several analysis scenarios.

More favorable evidence came from Meehan and colleagues, who evaluated coding accuracy, severity of illness, and process-based quality of care in Connecticut hospitals. ⁶⁶¹ Three process measures were selected by an expert panel based on medical literature and local practice patterns: 1) administration of thrombolytic therapy, 2) discharged on aspirin if no contraindication, and 3) discharged on a beta blocker if no contraindication. The hospitals with the highest risk-adjusted mortality had significantly lower utilization of beneficial therapies than the other hospitals in the sample. Although the Medicare Prospective Payment System Quality of Care study did not focus on specific therapeutic interventions, it also demonstrated significantly higher risk-adjusted mortality rates (using risk factors derived by chart review) among hospitals with "poor" processes of care than among hospitals with "good" or "medium" processes of care (30.1% versus 22.0% and 23.9%, respectively). ⁶⁷⁶ Chen ¹⁴⁶ showed that the hospitals designated by US News and World Report as "America's Best Hospitals" in cardiology, based on risk-adjusted mortality (using APR-DRGs) and reputation among physicians, had lower risk-adjusted mortality (using clinical predictors) among Medicare patients (15.6% versus 18.3-18.6%) and used aspirin and beta blockers more often than hospitals that were not so designated. Similarly, major teaching hospitals in the same Medicare data set had 20% lower risk-adjusted 30-day mortality than nonteaching hospitals; about half of this difference was attributable to greater use of beneficial therapies. ⁶⁷⁷ In the RAND PPS Quality of Care study in 1990, patients with higher process scale scores for AMI demonstrated significantly lower risk-adjusted 30-day AMI mortality on four out of five subscales and on an overall process scale. ⁶⁷⁶ In the California Hospital Outcomes Project, hospitals with low risk-adjusted AMI mortality were more likely to give aspirin within 6 hours of arrival in the emergency room, more likely to perform cardiac catheterization and revascularization procedures within 24 hours, and more likely to give heparin to prevent thromboembolic complications. However, there were no differences between low and high-mortality hospitals in the use or timing of thrombolytic or beta blocker therapy. ⁶⁵⁹

These somewhat conflicting findings may relate to the general insensitivity of mortality rates to process measures. Mant and Hicks conducted a systematic review of the literature to estimate the effect sizes for therapies proven effective for AMI patients, based on clinical trials and meta-analyses. ⁴³ The therapies assessed were beta blockade, aspirin, fibrinolysis, and angiotensin converting enzyme inhibitors. Using the best estimates of effect size and the proportion of patients eligible for treatment, the authors calculated the absolute risk reduction for low and high baseline mortality situations, with a resulting range of 5.1% to 16.4%. Given this range, they simulated the number of patients required to detect differences in care using either a "perfect system" for risk-adjusted mortality or a process-based quality of care audit. Using the same population of AMI patients, the difference in lives lost was detectable with one year of data collection on mortality or only two weeks of data collection on process of care.

The widespread recognition of the exceptionally strong evidence base supporting specific processes of care for AMI patients has led to numerous professional guidelines, guideline implementation projects,^{678, 679} and regional and national quality improvement initiatives. Through its Cooperative Cardiovascular Project and Sixth Scope of Work, the Health Care Financing Administration has focused considerable attention on improving processes of care for AMI, as a way to improve mortality and other outcomes. Hospitals in the four pilot states involved in this project (AL, CT, IA, WI) significantly improved their performance on each process indicator between 1992 and 1995, and simultaneously achieved a greater reduction in 30-day mortality (19.9% to 17.6%) than hospitals in other states (19.6% to 18.2%). This finding suggests, but does not prove, that hospitals can lower their AMI mortality rates by improving adherence to evidence-based guidelines.

Fosters true quality improvement

In general, physicians and hospitals have little discretion in their decisions to admit AMI patients, so it seems unlikely that the use of this indicator would impede access to needed care. However, a few patients who fail to respond to, or are ineligible for, resuscitative efforts in the emergency room may not be admitted if there is pressure to reduce inpatient mortality. Although such practices might bias comparisons of risk-adjusted inpatient mortality across hospitals, they would be unlikely to compromise patient outcomes (as resuscitative measures that fail in an emergency room would also fail in a coronary care unit). It is conceivable that patients could be discharged early to die at home or in a nursing home, although this may be unlikely due to the acute nature of the condition. Patient transfers to other hospitals will also have a greater effect on inpatient mortality rates, as noted in the OSHPD study, because hospitals vary widely in their transfer rates. Typically, 30-day overall mortality rates and 30-day inpatient mortality rates have been considered more valid than inpatient mortality rates based only on the initial hospitalization for AMI. The rank correlation between standardized AMI mortality measures based on inpatient deaths and measures based on 30-day deaths (at the hospital level) was 0.79 in a study of Medicare data. ²⁸⁹ This finding suggests that changes in length of stay may modestly alter the ranking of hospital performance using this measure.

Prior use

Inpatient AMI mortality, based on administrative data, has recently been used as a hospital quality indicator by the University Hospital Consortium, ³⁷⁰ the California Hospital Outcomes Project, ⁶⁸⁰ HealthGrades.com, ³⁷⁷ the Michigan Hospital Association (aggregated with congestive heart failure and angina), ³⁷³ and the Greater New York Hospital Association. ³⁷² In addition, the following organizations have used this indicator with risk-adjustment based on clinical data obtained through review of medical records: the Pennsylvania Health Care Cost Containment Council ⁶⁸¹ and Cleveland Health Quality Choice. ³⁷⁴ The Joint Commission for the Accreditation of HealthCare Organizations has adopted AMI mortality (from the MEDSTAT Corporation) as one of its core hospital performance measures. ⁴⁴³ AMI mortality is also a High-Level Performance Indicator for the United Kingdom's National Health Service. ⁵⁹⁰

Precision

This indicator is precise, with a raw provider level mean of 24.4% and a standard deviation of 16.1%. The systematic provider level standard deviation is high, at 3.4%. The provider level variation also accounts for a high percentage of total variation, at 0.8%. This means that relative to other indicators, a higher percentage of the variation occurs at the provider level, rather than the discharge level, although more of the variation occurs at the discharge level than some indicators. The signal ratio is only moderate, at 42.8%. This means that it is likely that the some of the observed differences in provider performance do not represent true differences in provider performance. The moderate R-square (59%) reflects the higher proportion of signal that can be extracted using multivariate techniques.

Bias

Signal variance decreases by over 25% with risk adjustment, indicating that some of the true variation among providers reflects differences in patient characteristics. The indicator performs fairly on the multiple measures of minimum bias. The rank correlation is fair at 0.747. The impact on the extremes is large. Only 36.3% of providers in the highest decile remain, and only 67.3% in the lowest decile remain, after risk adjustment. Similarly, the number of providers moving at least two deciles in relative rank is also high. The absolute magnitude of risk adjustment is also substantial.

Construct validity

AMI mortality does not load substantially on any of the three extracted factors. However, AMI mortality is correlated with several other indicators, including Bi-lateral catheterization (r=-.16, p<.0001), mortality for CHF (r=.46, p<.0001), pneumonia (r=.46, p<.0001), CABG (r=.50, p<.0001), stroke (r=.40, p<.0001), and GI hemorrhage (r=.38, p<.0001).

Face validity

Admission for heart failure is common with 400,000 new cases ⁶⁸² and 274,000 deaths ⁶⁸³ each year in the United States. Approximately 2 million persons in the U.S. have heart failure and this number will increase as the population ages. ⁶⁸² Population data from the United States have not indicated a decline in mortality over the last 20 years. ⁶⁸⁴ However, a recent study from Scotland has suggested that case-fatality mortality rates have improved between 1986 and 1995. ⁶⁸⁵

The accuracy of ICD-9 coding for heart failure has been questioned. Although the specificity of a principal diagnosis of heart failure is high (>95) the sensitivity is low. ⁶⁸⁶ Even when the principal or secondary diagnoses are used the sensitivity is only 63%, and the positive predictive value is 83.5% ⁶⁸⁶ Others have found lower positive predictive values (62.5%) but higher sensitivities (89.9%) for the combined use of principal and secondary diagnoses. ⁶⁸⁷ Face validity will be maximized by limiting analyses to patients with a principal diagnosis of heart failure.

Precision

Rates of short-term mortality vary from 6% ⁶⁸⁸ to 13%, ⁶⁸⁹ 14%, ⁶⁹⁰ 19.9%. ⁶⁸⁵

In-hospital survival appears to have improved recently. The 3-day mortality rate decreased by 41% in a study of 29,500 elderly patients in Oregon from 1991 to 1995. ⁶⁹¹ These data suggest that hospitals have improved care for heart failure patients.

There is substantial variation in hospital survival. In an analysis of 6 hospitals the survival ranged from 3.6% to 11.3%; however, no hospital had a statistically significantly higher rate. ⁶⁹²

Minimum bias

Mortality from heart failure is greatly influenced by patient characteristics. Mortality has been reported to be higher in older patients, males and ⁶⁹³ and possibly whites. ⁶⁹⁴ Using administrative data, the c-statistic was 0.68 for logistic models of in-hospital survival using the Charlson comorbidity (non-specific) score and 0.78 for a heart failure specific score. ⁶⁹⁵ The variables associated with increased in-hospital mortality in the heart failure specific score were age (relative weight 1), transfer, ⁶⁸³ cerebrovascular disease, ⁶⁸³ chronic obstructive pulmonary disease, ⁶⁸³ hyponatremia, ⁶⁸³ other hydro-electrolytic disturbance, ⁶⁸³ metastatic disease, ⁶⁸³ moderate to severe renal disease, ⁶⁸⁵ ventricular arrhythmia, ⁶⁸⁷ mild liver disease, ⁶⁸⁷ malignancy, ⁶⁸⁷ hypotension and shock.^{694, 695}

Another multivariable model of in-hospital mortality using clinical data (blood pressure, electrocardiogram) demonstrated a c-statistic of 0.9. ⁶⁹² When six hospitals were evaluated using this clinical model, expected mortality varied from 4% to 9% with one hospital having a significantly lower predicted rate (P = .01). The observed-expected mortality ranged from -3.8% to +4.7% (all NS).

Construct validity

There were no studies that specifically examined the construct validity of in-hospital mortality from heart failure. We did identify several studies that provide information on the validity of this indicator. On a patient level processes of care have been shown to decrease mortality; it is unknown how implementing these processes of care would actually affect provider-level mortality rates.

Survival is known to be improved for patients with heart failure and low ejection fraction if they receive ACE inhibitors. ⁶⁹⁶ There is a wide variation in the use of ACE inhibitors during hospitalization. ⁶⁹⁷ A measure of left ventricular ejection fraction (e.g. echocardiography) is recommended to determine which patients have depressed ventricular function and would benefit from life-prolonging medical therapy. Measures of left ventricular function have also been found to vary widely within hospitals. ⁶⁹⁷ Use of echocardiography has been associated with more ACE inhibitor use and improved survival. ⁶⁹⁸ In their PPS Quality of Care study in 1990, RAND reported better process of care for CHF on all five subscales and overall. On four of five process subscales and on the overall scale, patients demonstrated significantly lower risk-adjusted CHF mortality. For patients with a poor overall process scale score (in the lowest 25%), 30-day risk-adjusted mortality was 19% while that for patients with medium process scale scores was 13% and for patients with high scale scores (in the highest 25%) 30-day risk-adjusted mortality was 11%. ⁶⁷⁶

Fosters true quality improvement

Risk adjusted measures of mortality may lead to an increase in coding of comorbidities. Patients may be discharged early to a lower level of care (nursing home) so that the death does not occur in-hospital. All in-hospital mortality measures may create perverse incentives to reduce hospital mortality by discharging patients earlier, and thereby shifting deaths to skilled nursing facilities or outpatient settings. This phenomenon may also lead to biased comparisons among hospitals with different mean lengths of stay. The rank correlation between standardized mortality measures based on inpatient deaths and measures based on 30-day deaths (at the hospital level) was 0.71 and 0.78 in studies of Medicare ²⁸⁹ and all-payer Cleveland ⁶⁹⁹ data, respectively. This finding suggests that changes in length of stay may modestly alter the ranking of hospital performance using this measure. However, Rosenthal et al. found noevidence that hospitals with lower in-hospital standardized mortality had higher (or lower) early post-discharge mortality.

Prior use

Mortality for congestive heart failure has been widely used as a quality indicator. In addition to the use in the literature listed above, HealthGrades.com, ³⁷⁷ the University Hospital Consortium ³⁷⁰ and the Greater New York Hospital Association ³⁷² have used this measure. The Maryland Hospital Association include this measure in their Maryland QI Project Indicator set. ³⁶⁹ Cleveland Health Quality Choice ³⁷⁴ includes CHF mortality in their cardiovascular care measure. Likewise, Michigan Hospital Association ³⁷³ includes CHF in an aggregated mortality measure.

Precision

This indicator is precise, with a raw provider level mean of 7.5% and a standard deviation of 9.5%. The systematic provider level standard deviation is moderate, at 2.1%. The provider level variation also accounts for a moderate percentage of total variation, at 0.7%. This means that relative to other indicators, a lower percentage of the variation occurs at the provider level, rather than the discharge level. The signal ratio is only moderate, at 53.5%. This means that it is likely that the some of the observed differences in provider performance do not represent true differences in provider performance. The moderate R-square (69.7%) reflects the higher proportion of signal that can be extracted using multivariate techniques.

Bias

Signal variance does not change with risk adjustment. The indicator performs fairly to well on the multiple measures of minimum bias. The rank correlation is good at 0.794. The impact on the extremes is large. Only 39.8% of providers in the highest decile remain, and only 72.9% in the lowest decile remain, after risk adjustment. Similarly, the number of providers moving at least two deciles in relative rank is also high. The absolute magnitude of risk adjustment is moderate.

Construct validity

Congestive heart failure mortality loads on factor 1. It is positively related to other medical mortality indicators, such as pneumonia, GI hemorrhage, and stroke, and to a lesser extent, hip fracture.

Face validity

Admission for gastrointestinal hemorrhage is fairly common (100/100,000 adults). Mortality rates for hemorrhage vary greatly, and lower mortality has been associated with more use of treatments such as early endoscopy (within 24-48 hours of presentation), though the strength of this relationship has not been established, with some studies failing to find significant relationships (see construct validity section). Mortality rates in large population based databases have not changed since the 1940s, though, there have been increases in the ages and comorbidities of patients. ⁷⁰⁰

Precision

Rates of mortality in gastrointestinal hemorrhage vary from 0-29%, with most studies reporting rates of 3.5%-11%.^700-704

Minimum bias

Mortality from gastrointestinal hemorrhage is highly influenced by patient comorbidities and other factors complicating the bleed, as well as the nature and severity of the bleed itself, which all vary substantially across patients with the condition. One study noted that some endoscopic findings, hemodynamic characteristics, and comorbidities were highly predictive of life-threatening events. ⁷⁰⁵ The same study found at reassessment that the strongest predictors of life-threatening events were reoccurring bleeding (3-6 times risk of mortality) and unstable comorbid diseases. Mortality rates in patients with gastrointestinal hemorrhage have been shown to increase with age. In one study, the overall mortality, with the absence of comorbidity, was 4%, while patients under the age of 60 years experienced a mortality of 0.1%. Concurrent malignancy and organ failure increased the mortality rate for patients under 60 years to 0.8%. ⁷⁰⁴

Patients who develop bleeding in-hospital have higher mortality rates than patients admitted with gastrointestinal bleeding that began outside the hospital. One study reported a difference in crude mortality rates of 33% versus 11%.^{702, 704}

One study tested the effect of risk-adjustment on hospital ranking for gastrointestinal hemorrhage mortality. Risk-adjusting for age, shock, and comorbidity (characteristics that are often reported on discharge abstracts) changed 30 hospitals rankings by more than 10. Adding diagnosis, endoscopy findings, and rebleed status changed 32 hospital rankings by more than 10. ⁷⁰⁰

Construct validity

We located no studies explicitly evaluating the construct validity of this indicator. On a patient level processes of care have been shown to decrease mortality. However, it is unknown how implementing these processes of care would actually affect provider-level mortality rates.

A number of medical treatments have been shown to be associated with bleeding control, though evidence on association with mortality is more limited. Endoscopy has been shown inconsistently to be associated with mortality. One meta-analysis showed a slight advantage for early endoscopy, ⁷⁰⁶ while another study found that endoscopy was not related to mortality in either the bivariate or multivariate analyses. ⁷⁰¹

Many of the deaths reported are not associated with bleeding per se. One study found that only one death was related to bleeding, and that patient had several severe comorbidities. ⁷⁰² Thus, in many cases, the deaths in patients with a diagnosis of gastrointestinal hemorrhage are probably not actually due to the bleed itself.

Fosters true quality improvement

Risk-adjusted measures of mortality may lead to an increase of coding of comorbidities or upcoding of diagnoses. All in-hospital mortality measures may create perverse incentives to reduce hospital mortality by discharging patients earlier, and thereby shifting deaths to skilled nursing facilities or outpatient settings. This phenomenon may also lead to biased comparisons among hospitals with different mean lengths of stay. We found no published evidence about whether the difference between inpatient and 30-day mortality for in GI hemorrhage is substantial enough to cause concern.

Prior use

GI hemorrhage is currently used by the Cleveland Choice Health Quality Choice ³⁶⁹ and the Maryland Hospital Association (in the Maryland QI Project). ³⁶⁹ GI hemorrhage is also included in the Michigan Hospital Association's aggregated mortality measure. ³⁷³

Precision

This indicator is precise, with a raw provider level mean of 4.6% and a standard deviation of 5.7%. The systematic provider level standard deviation is moderate, at 1.1%. The provider level variation also accounts for a moderate percentage of total variation, at 0.3%. This means that relative to other indicators, a lower percentage of the variation occurs at the provider level, rather than the discharge level. The signal ratio is low, at 20.2%. This means that it is very likely that the some of the observed differences in provider performance do not represent true differences in provider performance. The moderate R-square (55.5%) reflects the higher proportion of signal that can be extracted using multivariate techniques.

Bias

Signal variance decreases by more than 25% with risk adjustment, suggesting that some of the observed variance is due to differences in patient characteristics. The indicator performs fairly to well on the multiple measures of minimum bias. The rank correlation is good at 0.803. The impact on the extremes is large. Only 48.9% of providers in the highest decile remain, and only 32.2% in the lowest decile remain, after risk adjustment. Similarly, the number of providers moving at least two deciles in relative rank is also high. The absolute magnitude of risk adjustment is moderate.

Construct validity

GI hemorrhage mortality loads on factor 1. It is positively related to mortality indicators, such as pneumonia, stroke, and congestive heart failure, and to a lesser extent, hip fracture.

Face validity

Hip fractures are a common cause of morbidity and functional decline among elderly persons. In addition, hip fractures are associated with a significant increase in the subsequent risk of mortality, which persists for a minimum of 3 months among the oldest and most impaired individuals at baseline,^{707, 708} and perhaps up to several years among younger and less impaired individuals.^{709, 710} Fractures of the femoral neck or intertrochanteric region are usually caused by minimal trauma (e.g., fall on a step, on a level surface, or from a chair) in the setting of osteoporosis, a common condition characterized by demineralization and weakening of weight-bearing bones. About 89% of hip fracture patients are elderly, and they often have multiple comorbidities and pre-fracture functional impairments. As a result, they are at significant risk of such postoperative complications as pneumonia, myocardial ischemia, arrhythmias, and deep vein thrombosis. If these complications are not recognized and effectively treated, life-threatening problems such as respiratory failure, myocardial infarction, and pulmonary embolus may ensue.

Precision

Hip fracture is the most common type of fracture requiring hospitalization; about 382,000 discharges in 1998 listed a diagnosis of hip fracture, of which 329,000 listed it as the principal diagnosis (12.0 per 10,000 persons). ⁷¹¹ Based on the all-payer database in California, each hospital admitted an average of 51.6 elderly hip fracture patients requiring surgical repair per year in 1995-96. ⁷¹² The largest published study of in-hospital mortality reported a rate of 4.9% in 1979-88. ⁷¹³ These data suggest that mortality rates are likely to be relatively reliable at the hospital level.

Minimum bias

There is relatively little potential for selection bias, because almost all patients with hip fracture are hospitalized. However, hip fracture patients may tend to be sicker at some hospitals than at others, due to variations in health status across the catchment areas of different facilities. The known predictors of in-hospital or 30-day mortality can be divided into several categories. Demographic predictors include age, male sex, and prior residence in a nursing home. Comorbidity predictors include malnutrition, venous disease, digestive diseases, cardiovascular diseases (including congestive heart failure, angina, and atrial fibrillation), neoplasms, disorientation or delirium, chronic obstructive pulmonary disease, prior hospitalization within one month, the Charlson comorbidity index (or the number of chronic medical conditions), and the ASA (American Society of Anesthesiology) physical status score. The only proven functional predictor of short-term mortality is dependency in any activity of daily living. Finally, fracture site may be a significant predictor, although probably more for long-term outcomes than for short-term outcomes. To the extent that these factors are more prevalent at some hospitals than at others, hip fracture mortality rates may be susceptible to confounding bias. In the absence of studies explicitly comparing models with and without clinical data elements, it is difficult to assess whether administrative data contain sufficient information to remove bias.

Construct validity

There is conflicting evidence on the construct validity of this indicator. The association between risk-adjusted mortality (using clinical data obtained by chart abstraction) and implicit and explicit process criteria was explored as part of RAND's Prospective Payment System Quality of Care study. Whereas Medicare patients with poor "process of care" had higher risk-adjusted 30-day mortality than those with good "process of care" for four medical conditions, there was no difference for hip fracture (4.6% versus 5.1%, RR=0.90). None of the process subscales (physician cognitive, nurse cognitive, technical diagnostic, technical therapeutic, monitoring) was associated with risk-adjusted 30-day mortality. ⁶⁷⁶ A more recent British study identified one East Anglican hospital with significantly lower than average risk-adjusted mortality (at 90 days); patients at this hospital "were routinely treated by a designated multidisciplinary team for fracture of the hip, with early assessment and surgery, much of which was performed by one surgeon," routine thromboembolism prophylaxis, and early mobilization. ⁷¹⁴

There is very little evidence supporting an association between hospital volume and mortality following hip fracture repair. (Following Halm, Lee, and Chassin ⁸⁰ , we did not find this evidence to be sufficiently strong to recommend total hip fracture volume as a separate volume indicator.) Using administrative data from Florida, without any risk-adjustment, Lavernia ⁷¹⁵ found no association between surgeon volume and in-hospital mortality. They did not report the effect of hospital volume, if any. A study of Medicare data from 1979 and 1980 showed no association with 60-day mortality, after adjusting for age, sex, and region, ⁷¹⁶ although a more recent study of 1988 and 1990 Medicare data found higher than expected in-hospital mortality at low-volume hospitals. ⁷¹⁷ Maerki ⁷¹⁸ and Luft ²³⁹ found "no clear pattern" in the relationship between hospital volume and hip fracture mortality, using 1972 data from the Commission on Professional and Hospital Activities (CPHA), although there was a suggested effect among low risk patients.^{719, 720} Hughes and colleagues ⁵² used 1982 data from CPHA and did find a significant association between volume and inpatient mortality in a hospital-level regression analysis. These inconsistencies provide limited support for the construct validity of mortality as a quality indicator.

There is also substantial evidence that at least two major causes of death among hip fracture patients are partially preventable. Perez et al. ⁷²¹ reviewed 581 autopsy reports on hip fracture patients who died in a single British hospital between 1953 and 1992, and reported that 80 (14%) and 55 (9%) deaths were attributable to pulmonary emboli and acute myocardial infarction, respectively. The most common cause of death after hip fracture was bronchopneumonia (46%); no medical intervention has been shown to reduce the incidence of, or mortality due to, this complication. Thromboembolic prophylaxis using either unfractionated or low molecular weight heparin reduces the incidence of radiographically documented deep vein thrombosis from 39% to 24% (OR=0.41, 95% CI=0.31-0.55), whereas physical devices reduce the incidence from 19% to 6% (OR=0.24, 95% CI=0.13-0.44). ⁷²² Although meta-analysis suggests a reduction infatal pulmonary emboli with heparin prophylaxis (OR=0.39, 95% CI 0.14-1.09), there are insufficient data to draw firm conclusions regarding symptomatic emboli. A recent controlled trial suggests that aspirin reduces the incidence of symptomatic deep vein thrombosis (RR=0.71, 95% CI=0.52-0.97), symptomatic pulmonary emboli (RR=0.57, 95% CI=0.40-0.82), and fatal pulmonary emboli (RR=0.42, 95% CI=0.24-0.73) after hip fracture. ⁷²³ These experimental data are supported by population-based observational data from at least two areas.^{714, 724} One randomized controlled trial that included high-risk patients undergoing noncardiac surgery suggested that perioperative use of beta blockers may reduce the incidence of postoperative AMI. ⁷²⁵ A recent meta-analysis reported a nonstatistically significant reduction in AMI (RR=0.70, 95% CI=0.64-3.57) and a marginally significant reduction in total mortality at one month (RR=0.72, 95% CI=0.51-1.00) with regional anesthesia.^{726, 727} Nutritional supplementation may be a useful strategy to prevent postoperative complications, but no effect on mortality has been demonstrated. ⁷²⁸ Finally, several aspects of surgical technique may be associated with higher short-term mortality, including the use of hemiarthroplasty instead of internal fixation for displaced femoral neck fractures,^729-732 the use of cement ⁷³¹ or a posterior approach ⁷³² in hemiarthroplasty for such fractures, and delayed surgical fixation.^{729, 730, 733-735} However, these findings come from observational studies that are susceptible to bias. The deleterious effect of hemiarthroplasty disappears with longer follow-up.^{732, 736} Nonetheless, these studies give providers with high hip fracture mortality rates some ideas for investigation or intervention.