Phase 1

To identify potential indicators, we performed a structured review of the literature. Using Medline, we identified the search strategy that returned a test set of known applicable articles in the most concise manner. The final Medical Subject Heading (MeSH) terms used were "hospital, statistic and methods" and "quality indicators." This search resulted in approximately 2600 articles. These articles were screened for relevancy to this project according to specified criteria. The yield from the search and screen was 181 relevant articles.

Information from these articles was abstracted in two stages by clinicians, health services researchers and other team members. In the first stage, preliminary abstraction, we evaluated each of the 181 identified articles for the presence of a defined quality indicator, potential quality indicators, and obvious strengths and weaknesses. To qualify for full abstraction (stage 2 of phase 1), the articles must have explicitly defined a novel quality indicator. Similar to previous attempts to cull new indicators from the peer reviewed literature, few articles (27) met this criterion. Information on the definition of the quality indicator, validation and rationale were collected during full abstraction (stage 2 of phase 1).

Additional potential indicators were identified using the CONQUEST (COmputerized Needs-oriented QUality Measurement Evaluation SysTem) database, the National Library of Healthcare Indicators (developed by JCAHO), a list of ORYX™ approved indicators provided by the Joint Commission on Accreditation of Healthcare Organizations (JCAHO), and from the telephone interviews.

Phase 2

After identifying over 200 potential indicators during Phase 1, we evaluated the indicators. Initially, team members evaluated the clinical rationale of each indicator, and selected the most promising indicators based on certain criteria, including frequency of the event and sound clinical rationale. HCUP I (Healthcare Cost and Utilization Project) indicators were not evaluated in this stage and were automatically selected for the next step of evaluation. Second, those indicators passing the initial screen (including the HCUP I indicators) were subjected to basic empirical tests of precision, as described below. Third, a full literature review was conducted for those indicators with adequate performance on the precision tests. Medline was searched for articles relating to each of the six areas of evaluation, described in the evaluation framework. Clinicians, health services researchers and other team members searched the literature for evidence, and prepared a referenced summary description of the evidence from the literature on each indicator. These indicators also underwent full empirical analyses (see Empirical Methods).

Data set

The primary data sets used were the HCUP Nationwide Inpatient Sample (NIS) and the State Inpatient Database (SID) for 1995-1997. The annual NIS consists of about 6,000,000 discharges and over 900 hospitals from participating States. The SID contains all discharges for the included states. Most of the statistical tests used to compare candidate indicators were calculated using the SID, because the provider level results were similar to the NIS, and the SID includes all discharges for the direct calculation of area rates.

Precision

The first step in the analysis involved precision tests to determine the reliability of the indicator for distinguishing real differences in provider performance. Any quality indicator consists of both signal (`true' quality, that is what is intended to be measured) and noise (error in measurement due to sampling variation or other non-persistent factors). For indicators that may be used for quality improvement or other purposes, it is important to know with what precision, or surety, a measure can be attributed to an actual construct rather than random variation. For some indicators, the precision will be quite high for the raw measure. For other indicators, the precision will be rather low. However, it is possible to apply additional statistical techniques to improve the precision of these indicators. These techniques are called signal extraction, and are designed to "clean" or "smooth" the data of noise, and extract the actual signal associated with provider or area performance. We used two techniques for signal extraction to potentially improve the precision of an indicator. Detailed methods are contained in the full text of the methods section. First, univariate methods estimated the "true" quality signal of an indicator based on information from the specific indicator and one year of data. Second, new multivariate signal extraction (MSX) methods estimated the signal based on information from a set of indicators and multiple years of data. In most cases, MSX methods extract additional signal.

For each indicator, the variance can be decomposed into three components: variation within a provider (i.e. differing performance due to differing patient characteristics), variation among providers (i.e. actual differences in performance among providers), and random variation. An ideal indicator would have a substantial amount of the variance explained by between-provider variance, which might possibly result from differences in quality of care, and a minimum amount of random variation.

We conducted four tests of precision (signal standard deviation, provider/area variation share, signal-to-noise ratio and the in-sample r-squared) in order to estimate the magnitude of between-provider variance on each indicator.

(1) Signal Standard Deviation: For the first test of precision, we calculated the signal standard deviation of the QI, which measures the extent to which performance of the QI varies systematically across hospitals or areas. The signal standard deviation is the best estimate of the variation in QI performance that appears to be systematic - truly associated with the hospital or area, and not the result of random variations in patient case mix or environmental conditions. Because many patient characteristics and random events may influence performance of a QI, only some of the apparent variation in performance at the hospital or area level is likely to be systematic. Systematic variation will be larger to the extent that: sample sizes are larger (allowing more precise estimation of the true effect), and patient-associated variation is smaller (allowing more precise estimates for a given sample size). Large systematic variation across hospitals and areas suggests that there are opportunities for further study and quality improvement. By contrast, if the signal standard deviation is small, then there may be little gained by improving the performance of lower-ranked hospitals or areas to the levels achieved by the higher-ranked hospitals or areas.

(2) Provider/Area Share:

Another precision test we calculated is the percentage of signal variance relative to the total variance of the QI. In general, the variance in a quality indicator can be divided as follows:

Total Variance = Patient-Level Variance + Provider-Level Variance.

Typically, most variation in the QIs (e.g., whether or not a vaginal birth occurred for a patient with a prior C-section) is associated with patient-level factors, and may have nothing whatsoever to do with providers or areas. In turn, the apparent Provider-Level Variance (or analogously, Area-Level) can be divided into two components:

Provider-Level Variance = Provider-Level Signal Variance + Noise Variance.

That is, some of the variance in a QI rate for a particular provider (adjusted or unadjusted) is a reflection of random chance ¹¹: in a given year, some hospitals or areas will by chance have more or fewer occurrences of the numerator related to a particular QI. The remaining variance (signal variance), and only the remaining variance, is attributable to systematic or true differences in the QI across providers. The percentage of signal variance relative to total variance will be larger to the extent that: within-hospital patient variance is small; sample sizes are larger; and true differences in performance across hospitals are larger. Unlike the signal standard deviation measure of precision, the percentage of signal variance is a relative measure: relative to the total variation (patient plus provider) in the QI, what fraction appears to be associated with systematic provider differences? Thus, higher percentages suggest enough differences across hospitals or areas that true differences across providers or areas can be distinguished from other "random" influences on QI performance.

(3) Signal-to-Noise Ratio:

The third precision measure we calculated is the Signal-to-Noise ratio in the provider- or area- level variation in the QI measure. This measure answers the question: of the apparent variation in QIs across providers, what fraction appears to be truly related to systematic differences across providers, and not random variations ("noise") from year to year? As noted earlier, the provider-level variation includes both signal variation and random variations in the QI measures across providers ("noise"). In other words, it is the "signal" to "signal plus noise" ratio. As such, the Signal-to-Noise ratio measure is somewhat misnamed. Its definition is:

Signal-to-Noise Ratio = (Provider-Level Signal Variance)/(Provider-Level Total Variance).

In general, if a QI's signal-to-noise ratio is high, then it is likely that apparent variations in performance across providers are not the result of random chance, and careful attention to distinguishing true from random variation across providers will have little impact on the measured performance of a provider. Similarly, for indicators measuring area level rates, the aim is to distinguish between actual and apparent differences between areas.

(4) In-sample R-Squared:

The fourth precision measure identifies the incremental benefit of applying multi-variate signal extraction methods for identifying additional signal on top of the signal-to-noise ratio.

Bias

To provide empirical evidence on the sensitivity of candidate QIs to potential bias from differences in patient severity, we compared unadjusted performance measures for specific hospitals with performance measures that were adjusted for age, gender, and, where possible, patient clinical factors available in discharge data. We used the 3M APR-DRG System Version 12 with Severity of Illness and Risk of Mortality subclasses, as appropriate, for risk adjustment of the hospital quality indicators. Nonetheless, we recognize that this system is not ideal, because it provides only four severity levels within each base APR-DRG, omits important physiologic and functional predictors, and potentially misadjusts for iatrogenic complications. For a few measures, no APR-DRG severity categories were available, so that unadjusted measures were compared to age-sex adjusted measures. Because HCUP data do not permit the construction of area measures of differences in risk, only age-sex adjustment is feasible for the area-level indicators evaluated. We used a range of bias performance measures, most of which have been applied in previous studies.^{9-13, 17-19} We note that these comparisons are based entirely on discharge data. In general, we expect performance measures that are more sensitive to risk adjustment using discharge data also to be more sensitive to risk adjustment using more complete clinical data, though the differences between the adjusted and unadjusted measures may be larger in absolute magnitude than the discharge data analysis would suggest. However, there may not be a correlation between discharge and clinical-record adjustment. Specific cases where previous studies suggest a greater need for clinical risk adjustment are discussed in our literature reviews of relevant indicators. To investigate the degree of bias in a measure, we performed five empirical tests. Each test was repeated for the "raw" data, for data smoothed by univariate techniques (one year of data, one indicator), and for data smoothed by multivariate (MSX) techniques (using multiple years of data, all indicators).

1. Rank Correlation Coefficient: The Spearman correlation coefficient of the rank of the area/hospital without and with risk adjustment gives the overall impact of risk adjustment, on relative provider or area performance.
2. Average Absolute Value of Change Relative to Mean: The average absolute change in the performance measure of the hospital or area with and without risk adjustment, normalized by the average value of the QI. Thus, it is also a relative measure that theoretically can range from zero (no change) to a much higher value. This measure highlights the amount of absolute change in performance, without reference to other providers' performance. This does not rely on the distribution of performance on this indicator, as relative performance measures do.
3. Percentage of High That Remains High Decile, or
4. Percentage of Low Decile That Remains in Low Decile: These two measures report the percentage of hospitals or areas that are in the highest and lowest performance deciles without risk adjustment, that remain there with risk adjustment. A measure that is insensitive to risk adjustment should have rates of 100%, whereas measures for which risk adjustment affects the top- and bottom-performers substantially should have much lower percentages.
5. Percentage That Change More Than Two Deciles: the percentage of hospitals whose relative rank changes by a substantial distance - more than 20% - with and without risk adjustment. Ideally, risk adjustment would not have a substantial effect for very many hospitals.

Construct validity

Two measures of the same construct would be expected to yield similar results. If quality indicators do indeed measure quality, at least in a specified domain, such as ambulatory care, one would expect measures to be related. As quality relationships are likely to be complex, and outcomes of medical care are not entirely explained by quality, perfect relationships between indicators seem unlikely. Nonetheless, we would expect to see some relationships between indicators.

To measure the degree of relatedness between indicators, we conducted a factor analysis, a statistical technique used to reveal underlying patterns among large numbers of variables. The output for a factor analysis is a series of "factors" or overarching constructs, for which each indicator would "load" or have a relationship with others in the same factor. The assumption is that indicators loading strongly on the same factor are related to each other via some independent construct. We used an orthogonal rotation to maximize the possibility that each indicator would load on one factor only, to ease the interpretation of the results. In addition to the factor analysis, we also analyzed correlation matrices for each type of indicator (provider level, ambulatory care sensitive condition (ACSC) area level, and utilization area level).

The construct validity analyses provided information regarding the relatedness or independence of the indicators. Such analyses cannot prove that quality relationships exist, but they can provide preliminary evidence on whether the indicators appear to provide consistent evidence related to quality of care. For hospital volume quality indicators, we evaluated correlations with other volume and hospital mortality indicators, to determine whether the proposed HCUP II indicators suggested the same types of volume-outcome relationships as have been demonstrated in the literature.

Results of empirical evaluations

Statistical test results for candidate indicators were compared. First, the results from precision tests were used to sort the indicators. Those indicators performing poorly were eliminated. Second, the results from bias tests were conducted to determine the need for risk adjustment. Finally, construct validity was evaluated to provide some evidence on the nature of the relationship between potential indicators.

Phase 1: Identification of Indicators

Phase 2: Evaluation of Indicators

The result of Phase 1 was a list of potential indicators with varied information on each depending on the source. Since each indicator relates to an area that potentially screens for quality issues, we developed a structured evaluation framework to determine measurement performance. A series of literature searches were then conducted to assemble the available scientific evidence on the quality relationship each indicator purported to measure. Due to limited resources, we could not review all of the indicators we identified in Phase 1, and therefore selected some for detailed review using the evaluation framework. The criteria used to select these indicators are described later.

Step 2. Identification of the evidence

We searched the literature for evidence in each of the six areas of indicator performance described above, and in the clinical areas addressed by the indicators. The search strategy used for Phase 2 began with extensive electronic searching of MEDLINE, ¹⁷⁸ PsycINFO, ¹⁷⁹ and the Cochrane Library. ^{180 3 3} Thus, in contrast to conducting systematic reviews of purely clinical topics, we reasoned that the European literature not captured in the Medline database or Cochrane Library would almost certainly represent studies of questionable relevance to the US health system.

The extensive electronic search strategy involved combinations of MeSH terms and keywords pertaining to clinical conditions, study methodology, and quality measurement (see Figure 2).

Figure

Figure 1. Abbreviated diagram of literature review.

Figure

Figure 2. Example search strings.

Figure

Figure 3. Example search. All searches included Limits: Publication Date from 1990 to 2000 Language, English. The results of searches 5 and 11 were scanned (titles and abstracts) to pull relevant studies, and the bibliographies of (more...)

Additional literature searches were conducted using specific measure sets as "keywords". These included "Maryland Quality Indicators Project," "HEDIS and low birth weight, or cesarean section, or frequency, or inpatient utilization," "IMSystem," "DEMPAQ," and "Complications Screening Program."

We also searched the bibliographies of key articles, and hand searched the Tables of Contents of general medical journals, as well as journals focusing in health services research or in quality measurement. This list of journals included Medical Care, Health Services Research, Health Affairs, Milbank Quarterly, Inquiry, International Journal for Quality in Healthcare, and the Joint Commission Journal on Quality Improvement.

These literature searches and on-line screening for relevancy retrieved over 2000 additional articles, which were added to the project database. These articles were used for evaluations of individual indicators. Those articles cited are listed in the references.

The use of medical literature databases likely eliminated much of the "gray literature" that may be applicable to this study. Given the limitations and scope of this study, we did not complete a formal search of the "gray literature" beyond that which was previously known by the project team, or resulted from our telephone interviews.

Risk Adjustment Literature Review Methods

Our literature review for risk adjustment of the HCUP Quality Indicators combined evaluation criteria common to evidence studies on the performance of risk adjustment systems with additional considerations of importance to the potential HCUP users. These considerations were determined through semi-structured interviews with users, discussed earlier in this report. In general, users viewed risk adjustment as an important component of the HCUP QIs refinement. State data organizations and agencies involved in reporting of hospital performance measures especially tended to view risk-adjustment as essential for the validity of the results and acceptance by participating hospitals. Concerns that patient severity differed systematically among providers, and that this difference might drive the performance results, was frequently mentioned as a reason for limited reporting and public release of the HCUP I QIs to date, especially for outcome-oriented measures like mortality following common elective procedures.

Risk Adjustment Empirical Methods

We used the APR-DRG system, with severity and risk of mortality classifications, in two ways:

• to evaluate the impact of measured differences in patient severity on the relative performance of hospitals and areas, by comparing QI measures with and without risk adjustment; and
• to risk-adjust the hospital- and area-specific measures.

As our previous review noted, the available literature on the impact of risk adjustment on indicator performance is limited, but suggests that at least in some cases different systems may give different results. Problems of incomplete or inconsistent coding across institutions are probably important contributing factors to the differences in results. Thus, definitive risk adjustment for some indicators may require detailed reviews of medical charts and additional data sources (charts may also be incomplete), just as definitive quality measures for many indicators may require additional sources of information. However, the importance of random variations in patients means that whatever risk adjustment and quality measurement system is chosen should be used in conjunction with statistical methods that seek to minimize other sources of noise and bias. Our empirical analysis is intended to illustrate the approach of combining risk adjustment with smoothing techniques, including suggestive evidence on the importance of risk adjustment for potential HCUP II QIs, using a risk adjustment system that can be implemented on discharge data by most HCUP users. We also supplement our empirical analysis with a review of the clinical literature, to identify additional clinical information that is important to consider for certain indicators. In particular, our literature review highlights a few indicators where risk adjustment with additional clinical data has been shown to be particularly important, and where important differences in case mix seem less likely to be related to the secondary diagnoses used to risk-adjust discharge data.

This section describes how we implement risk-adjustment using patient demographics (age and sex) along with the APR-DRG classification system. The next section describes statistical methods we employ to account for additional sources of noise and bias not accounted for by observed patient characteristics. By applying these methods to all of the potential HCUP II QIs, we are able to evaluate the relative importance of both risk adjustment and smoothing in terms of the relative performance of hospitals (or areas) compared to the "raw" unadjusted QIs based on simple means from NIS discharge data. The simple means fail to account both for differences in the indicators that are attributable to systematic differences in measured and unmeasured patient mix across hospitals/areas that are measured in the discharge data, and for random variations in patient mix. We adopted a multivariate regression approach to adjust performance measures for measured differences in patient mix, which permits the inclusion of multiple patient demographic and severity characteristics.

Specifically, if we denote whether or not the event associated with a particular indicator Y^k (k=1,…,K) was observed for a particular patient i at hospital/area j (j=1,…,J) in year t (t=1,…,T), then the regression to construct a risk-adjusted "raw" estimate a hospital/area's performance on each indicator can be written as:

(1) , where

Y^k _ijt is the k^th quality indicator for patient i discharged from hospital/area j in year t (i.e., whether or not the event associated with the indicator occurred on that discharge);

M^k _jt is the "raw" adjusted measure for indicator k for hospital/area j in year t (i.e., the hospital/area "fixed effect" in the patient-level regression);

Z_ijt is a vector of patient covariates for patient i discharged from hospital/area j in year t (i.e., the patient-level measures used as risk adjusters);

Π^k _t is a vector of parameters in each year t, giving the effect of each patient risk adjuster on indicator k (i.e., the magnitude of the risk adjustment associated with each patient measure); and

^k _ijt is the unexplained residual in this patient-level model.

The hospital/area specific intercept M^k _jt is the "raw" adjusted measure of a hospital/area's performance on the indicator, holding patient covariates constant. In most of the empirical analysis that follows, the patient-level analysis is conducted using data from all hospital/areas. (The model shown implies that each hospital/area has data for all years, and within each year has data on all outcomes; however, this is not essential in order to apply risk adjustment methods.)

We estimated these patient-level regressions by linear ordinary least-squares (OLS). In general, the dependent variables in the regressions are dichotomous, which raises the question of whether a method for binary dependent variables such as logit or probit estimation might be more appropriate. However, previous work by McClellan and Staiger has successfully used OLS regression for similar analyses of hospital/area differences in outcomes. In addition, estimating logit or probit models with hospital/area fixed effects cannot be done with standard methods; it requires computationally intensive conditional maximum likelihood methods that are not easily extended to multiple years and multiple measures. ¹⁸⁵ A commonly-used "solution" to this problem is to estimate a logit model without hospital/area effects, and then to use the resulting predictions as estimates of the expected outcome. However, this method yields biased estimates and predictions of hospital performance. In contrast, it is easy to incorporate hospital/area fixed effects into OLS regression analysis, the resulting estimates are not biased, and the hospital/area fixed effects provide direct and easily-interpretable estimates of the outcome rate for a particular hospital/area measure in a particular year, holding constant all observed patient characteristics.

Of course, it is possible that a linear probability model is not the correct functional form. However, as in our earlier work, we specified a very flexible functional form, including full interactions among age and sex covariates as well as a full set of APR-DRG risk adjusters. In our sensitivity analyses for selected quality measures, this flexible linear probability model produced estimates of the effects of the risk adjusters that did not differ substantially from nonlinear (logit and probit) models. Another potential limitation of the OLS approach is that it may yield biased estimates of confidence intervals, because the errors of a linear probability model are necessarily heteroskedastic. Given the large sample sizes for the parameters estimated from these regressions (most indicators involve thousands of "denominator" discharges per year), such efficiency is not likely to be an important concern. Nevertheless, we estimated models using Weighted Least Squares ¹⁸⁶ to account for heteroskedasticity, to see if our estimates were affected. We obtained very similar estimates of adjusted indicator performance.

Specifically, in addition to age, sex, and age*sex interactions as adjusters in our model, we also included the APR-DRG category for the admission and the APR-DRG constructed severity subclass (or risk-of-mortality subclass for mortality measures). APR-DRGs are a refinement of the DRGs used by the Health Care Financing Administration, with additional classifications for non-Medicare cases (e.g., neonates). The severity subclass evaluates the episode of care on a scale of 1 (minor) to 4 (extreme). In the APR-DRG Version 12, Severity of Illness is defined as the "extent of physiologic de-compensation or organ system loss of function." The APR-DRG severity of illness subclass was designed principally to predict resource use, particularly length-of-stay. As such, because this risk-adjustment system was not designed to predict utilization rates, for example, in our evaluation of each indicator we do not consider lack of impact of risk-adjustment to be evidence of lack of real bias. However, we do consider impact of risk-adjustment to be evidence of problems of potential bias. The literature review further informs us to potential sources of bias, and the prior use of each indicator may require collection of supplemental data for confounding clinical conditions. We discuss these issues further in the Results Chapter.

For each indicator, we excluded from our risk adjustment model the APR-DRG groupings in the Major Diagnostic Category (MDC) related to that indicator. The groupings are either medical (based on diagnoses) or surgical (based on procedures), and we excluded groupings in the MDC of the same type. For example, for the Coronary Artery Bypass Graft rate indicator, we excluded all surgical APR-DRGs in MDC `05' (`Diseases and Disorders of the Circulatory System'). For GI Hemorrhage mortality, we excluded all medical APR-DRGs in MDC `06' (`Diseases and Disorders of the Digestive System'). Some of the indicators fall into only a few DRG categories. All discharges with carotid endarterectomy, for example, were within DRG `005', (`Extracranial Vascular Procedures'). For these indicators, we relied primarily on the severity subclass, which is independent of the DRG.

Actual implementation of the model involves running a regression with potentially a few thousand variables (each DRG divided into four severity subclasses) on millions of observations, straining the capacity of most statistical software and computer systems. In order to limit the number of covariates (DRG groups) in the model, we restricted the total number to 165 categories (DRG by severity), which was for all indicators sufficient to include 80% of discharges. We maintained all severity or risk-of-mortality subgroups for each APR-DRG included in the model in our construction of the raw adjusted estimates. The adjusted estimates of hospital performance are reported and used to compute descriptive statistics for each indicator in each year, as noted below. They are also used to construct smoothed estimates of each indicator.

The risk-adjusted estimates of hospital performance (age, gender, APR-DRG) and area performance (age, gender only) were used to construct descriptive statistics and smoothed estimates for each QI, as described in the next section.

Analysis Approach

Intuition Behind Univariate and Multi-variate Methods

An important limitation of many quality indicators is their imprecision, which complicates the reliable identification of persistent differences among providers in performance. The imprecision in quality indicators arises from two sources. The first is sampling variation, which is a particular problem for indicators based on small numbers of patients per provider (where the particular patients treated by the provider in a given year are considered a `sample' of the entire population who might have been treated or will be treated in the near future). The amount of variation due to the particular sample of patients is often large relative to the total amount of provider level variation that is observed in any given quality indicator. A second source of imprecision arises from non-persistent factors that are not sensitive to the size of the sample; for example, a severe winter results in higher than usual rates of pneumonia mortality. Both small samples and other one-time factors that are not sensitive to sample size can add considerable volatility to quality indicators. Also, it is not the absolute amount of imprecision that matters, but rather the amount of imprecision relative to the underlying signal (i.e., true provider level variation) that dictates the reliability of any particular indicator. Even indicators based on relatively large samples with no non-persistent factors at work can be imprecise if the true level of variation among providers is negligible.

Our approach to account for the imprecision or lack of reliability is a generalization of the idea of applying a "shrinkage factor" to each provider's estimate so that less reliable estimates are shrunk toward the national average. These `reliability-adjusted' estimates are sometimes referred to as `smoothed' estimates (because provider performance is less volatile over time) or `filtered' estimates (because the methods filter out the non-systematic noise, much like a radio filters our background noise to improve the radio signal). If the observed provider variation = signal variation + noise variation, then the shrinkage factor would be signal variation / (signal variation + noise variation). For example, suppose that the observed variation among providers in the in-hospital pneumonia mortality rate was a standard deviation of 10.2 percentage points, and the signal variation was a standard deviation of 5.0 percentage points. Then the shrinkage factor for this indicator is 0.240 = (0.050^2) / (0.102^2). The generalization of this approach seeks to extract additional signal using information on the relationship among multiple indicators over time.

Many of the key ideas behind the reliability-adjusted or filtered estimates are illustrated through a simple example. Suppose that one wants to evaluate a particular provider's performance based on in-hospital mortality rates among patients admitted with pneumonia, and data is available for the most recent two years. Consider the following three possible approaches: (1) use only the most recent mortality rate, (2) construct a simple average of the mortality rates from the two recent years, or (3) ignore the provider's mortality rate and assume that mortality is equal to the national average. The best choice among these three approaches depends on two important considerations: the signal-to-noise ratio in the provider's data, and how strongly correlated performance is from one year to the next. For example, suppose that the mortality rate for the provider was based on only a few patients, and one had reason to believe that mortality did not vary much across providers. Thenone would be tempted to choose the last option and ignore the provider's own data because of its low reliability (e.g. low signal-to-noise ratio). This is the idea of simple shrinkage estimators, in which less reliable estimates are shrunk toward the average for all providers. Alternatively, if one had reason to believe that provider mortality changed very slowly over time, one might choose the second option in hopes that averaging the data over two years would reduce the noise in the estimates by effectively increasing the sample size in the provider. Even with large numbers of patients one might want to average over years if idiosyncratic factors (such as a bad winter) affected mortality rates from any single year. Finally, one would tend to choose the first option, and rely solely on mortality from the most recent year, if such idiosyncratic factors were unimportant, if the provider admitted a large number of patients each year, and if mortality was likely to have changed from the previous year.

Our method of creating filtered estimates formalizes the intuition from this simple example. The filtered estimates are a combination of the provider's own quality indicator, the national average, and the provider's quality indicators from past years or other patient outcomes. As suggested by the example, to form the optimal combination one must know the amount of noise and signal variance in each indicator, as well as the correlation across indicators in the noise and signal variance. We estimate the noise variance (and covariance) in a straightforward manner for each provider, based on the number of patients on which each indicator is based. To estimate the signal variance (and covariance) for each quality indicator, we subtract the noise variance from the total variance observed in each indicator across providers (which reflects both signal and noise variance). In other words, the observed variation in quality indicators is sure to overstate the amount of actual variation across providers (because of the noise in the indicators). Therefore, we estimate the amount of true variation in performance based on how much the observed variation exceeded what would have been expected due to sampling error. Importantly, our method does not assume that provider performance is correlated from one year to the next (or that performance is correlated across indicators). Instead, we estimate these correlations directly from the data, and incorporate information from past years or other indicators only to the extent that these empirically estimated correlations are large.

Smoothed Estimates of Hospital Performance

For each hospital, we observed a vector of K adjusted indicator estimates over T years from estimating the patient-level regressions (1) run separately by year for each indicator as described in the preceding section. We know that each indicator is a noisy estimate of true hospital quality; in other words, it is likely that hospitals that performed especially well or badly on the measure did so at least in part due to chance. This fact is incorporated in Bayesian methods for constructing best-guess "posterior" estimates of true provider performance based on observed performance and the within-provider noise in the measures.

In particular, let M_j be the 1xTK vector of estimated indicator performance for hospital j. Then:

(2) M_j=μ_j + _j

Where μ_j is a 1xTK vector of the true hospital intercepts for hospital j, and _j is the estimation error (which has a mean zero and is uncorrelated with μ_j). Note that the variance of _j can be estimated from the patient-level regressions, since this is simply the variance of the regression estimates M_j. In particular, E( _jt' _jt) = Ω_jt and E( _jt' _js) = 0 for t ≠ s, where Ω_jt is the covariance matrix of the intercept estimates for hospital j in year t.

We wish to create a linear combination of each hospital's observed indicators in such a way that it minimizes the mean-squared prediction error. In other words, we would like to run the following hypothetical regression:

(3) μ^k _jt=M_jβ^k _jt + v^k _jt

but cannot do this directly, since μ is unobserved and the optimal β varies by hospital and year. While we cannot directly estimate equation (3), it is possible to estimate the parameters for this hypothetical regression. In general, the minimum mean squared error linear predictor of μ is given by M_j β, where β = [E(M_j'M_j)]⁻¹ E(M_j'μ_j). This best linear predictor depends on two moment matrices:

(4.1) E(M_j'M_j=E(μ_j' μ_j) + E( ^j' ^j)

(4.2) E(M_j'μ_j=E(μ_j' μ_j)

We estimate the required moment matrices directly as follows:

• Estimate E( _j' _j) with the patient-level OLS estimate of the covariance matrix for the parameter estimates M_j. Call this estimate S_j. Note that S_j varies across hospitals.
• Estimate E(μ_j' μ_j) by noting that E(M_j'M_j - S_j) = E(μ_j' μ_j). If we assume that E(μ_j' μ_j) is the same for all hospitals, then it can be estimated by the sample average of M_j'M_j - S_j. Note that it is easy to relax the assumption that E(μ_j' μ_j) is the same for all hospitals by calculating M_j'M_j - S_j for subgroups of hospitals.

With estimates of E(μ_j' μ_j) and E( _j' _j), one can form least squares estimates of the parameters in equation 3 which minimize the mean squared error. Analogous to simple regression, the prediction of a hospital's true intercepts is given by:

(5) = M_jE(M'_j M_j)^-1 E(M'_jμ_j) = M_j[E(μ'_jμ_j) + E (_jj)]^-1 E(μ'_jμ_j)

using estimates of E(μ_j' μ_j) and E( _j' _j) in place of their true values. One can use the estimated moments to calculate other statistics of interest as well, such as the standard error of the prediction and the r-squared for equation 3, based on the usual least squares formulas. We refer to estimates based on equation (5) as "filtered" estimates, since the key advantage of such estimates is that they optimally filter out the estimation error in the raw quality indicators.

Equation 5 in combination with estimates of the required moment matrices provides the basis for estimates of hospital quality. Such estimates of hospital quality have a number of attractive properties. First, they incorporate information in a systematic way from many outcome indicators and many years into the predictions of any one outcome. Moreover, if the moment matrices were known, our estimates of hospital quality represent the optimal linear predictors, based on a mean squared error criterion. Finally, these estimates maintain many of the attractive aspects of existing Bayesian approaches ⁸⁷ , while dramatically simplifying the complexity of the estimation. It is possible to construct univariate smoothed estimates of hospital quality, based only on empirical estimates for particular measures, using the models just described but restricting the dimension of M_j to only a particular indicator k and time period t. Of course, to the extent that the provider indicators are correlated with each other and over time, this will result in a less precise (efficient) estimate.

With many years of data, it helps to impose some structure on E(μ_j'μ_j) for two reasons. First, this improves the precision of the estimated moments by limiting the number of parameters that need to be estimated. Second, a time series structure allows for out-of-sample forecasts. We use a non-stationary, first-order Vector Autoregression structure (VAR). The VAR model is a generalization of the usual autoregressive model, and assumes that each hospital's quality indicators in a given year depend on the hospital's quality indicators in past years plus a contemporaneous shock that may be correlated across quality indicators. In most of what follows, we assume a non-stationary first-order VAR for m_jt (1xK), where:

(6)

Thus, we need estimates of the lag coefficient (Φ), the variance matrix of the innovations (Σ), and the initial variance condition (Σ), where Σ and Γ are symmetric KxK matrices of parameters and Φ is a general KxK matrix of parameters, for a total of 2K²+K parameters. For example, we must estimate 10 parameters for a VAR model with two outcomes (K=2).

The VAR structure implies that E(M_j'M_j - S_j) = E(m_j'm_j) = f(Φ,Σ,Γ). Thus, the VAR parameters can be estimated by Optimal Minimum Distance (OMD) methods, ¹⁸⁵ i.e. by choosing the VAR parameters so that the theoretical moment matrix, f(Φ,Σ,Γ), is as close as possible to the corresponding sample moments from the sample average of M_j'M_j - S_j. More specifically, let d_j be a vector of the non-redundant (lower triangular) elements of M_j'M_j - S_j, and let δ be a vector of the corresponding moments from the true moment matrix, so that δ=g(Φ,Σ,Γ). Then the OMD estimates of (Φ,Σ,Γ) minimize the following OMD objective function:
(7)
where V is the sample estimate of the covariance matrix for d, and is the sample average of d. If the VAR model is correct, the value of the objective function, q, will be distributed χ² (p) where p is the degree of over-identification (the difference between the number of elements in d and the number of parameters being estimated). Thus, q provides a goodness of fit statistic that indicates how well the VAR model fits the actual covariances in the data.

Finally, we use estimated R² statistics to evaluate the filtered estimates' ability to predict (in sample) and forecast (out-of-sample) variation in the true intercepts, and to compare our methods to conventional methods (e.g. simple averages, or univariate shrinkage estimators). If true hospital intercepts (μ) were observed, a natural metric for evaluating the predictions would be the sample r- squared:
(8)
where = μ_j - is the prediction error. Of course μ is not observed. Therefore, we construct an estimate using our estimate of E(μ_j' μ_j) for the denominator, and our estimate of E[(μ_j - )'(μ_j - )] for the terms in the numerator (where this can be constructed from the estimated moment matrices in equations 4.1-4.2). Finally, we report a weighted R-squared (weighting by the number of patients treated by each hospital).

As in earlier work using this method for cardiac care in the adult population, we validate our indicators using out-of-sample performance, based on forecasts (e.g., using the first two years of data to predict in subsequent year) and based on split-sample prediction (e.g., using one half of the patient sample to predict outcome indicators in the other half of the sample). For evaluating out-of-sample forecasts, we construct a modified R-squared of the forecast that estimates the fraction of the systematic (true) hospital variation in the outcome measure (M) that was explained:

where = M_j - is the forecast error, and S_j is the OLS estimate of the variance of the estimate M_j. This modified R-squared estimates the amount of variance in the true hospital effects that has been forecasted. Note that because these are out-of-sample forecasts, the R-squared can be negative indicating that the forecast performed worse than a naive forecast in which one assumed that quality was equal to the national average at all hospitals.

Empirical Analysis Statistics

Using the methods just described, we constructed a set of statistical tests to evaluate precision, bias, and construct validity. Each of the key statistical test results for these evaluation criteria was summarized and explained in the overview section of this chapter. Tables 1-3 provides a summary of the statistical analyses and their interpretation. Indicators were tested for precision first, and ones that performed poorly were eliminated from further consideration. These indicators are noted in Appendix 5. We assessed bias and construct validity for all recommended indicators. Cut-offs and further interpretation details are provided with results tables.

Table

Table 1. Precision Tests.

Table

Table 2. Bias Tests.

Table

Table 3. Construct Validity Tests.