Statistical Methods for Reducing Bias in Comparative Effectiveness Research When Using Patient Data from Doctor Visits

Charles E. McCulloch; John M. Neuhaus

doi:10.25302/6.2019.ME.130601466

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Statistical Methods for Reducing Bias in Comparative Effectiveness Research When Using Patient Data from Doctor Visits

Charles E. McCulloch and John M. Neuhaus.

Author Information and Affiliations

Washington (DC): Patient-Centered Outcomes Research Institute (PCORI); 2019 Jun.

Structured Abstract

Background:

Clinical databases, such as those collected in electronic health records systems, are increasingly being used for patient-centered outcomes research. Because the number and timing of visits in such data are often driven by patient characteristics and may be related to the outcomes being measured (which we call an outcome-dependent visit process), the danger is that this will result in biased and misleading conclusions compared with analyses from designed, prospective studies. Most, if not all, of the extant statistical methodology relies on unrealistic models for the outcome-dependent visit process.

Objectives:

Our goals were to (1) develop realistic models for outcome-dependent visit time models, (2) use theoretical calculations and simulation models to assess bias and efficiency in longitudinal statistical analyses applied to outcome-dependent visit databases, (3) provide guidance as to which types of statistical inferences are accurate and exhibit little bias when using databases collected with outcome-dependent visit times vs those that are likely to be inaccurate, and (4) make recommendations of how to deal with outcome-dependent visit processes in clinical research.

Methods:

We used semistructured interviews with clinician-scientists and analysis of their visit pattern data to develop realistic models for the connections between the likelihood of a visit and the outcome process. Using these realistic models, we assessed the performance of standard statistical methods, such as mixed-model regression, as well as analysis methods that purported to deal with outcome dependence. We used theory and simulations to evaluate the bias as well as the impact of including a small number of regularly scheduled visits. We used a wide variety of sensitivity analyses to determine the generality of the results. Using theory and simulation we also developed and evaluated the performance of methods to diagnose the outcome dependence.

Results:

Analysis approaches designed to deal with outcome dependence fared no better and, for some methods, worse than ignoring the outcome dependence and using standard, mixed-model analysis methods. The bias using standard methods under outcome-dependent visit processes is mostly confined to covariates that have associated random effects. A wide variety of sensitivity analyses confirm the generality of the results. To address this bias, we showed that inclusion of a few, non-outcome dependent observations can significantly reduce the bias when using maximum likelihood fitting methods. We also developed methods to diagnose the outcome dependence and showed that these diagnostic methods have high power to detect outcome-dependent visit processes. Furthermore, high power was achieved using these methods before maximum likelihood-based statistical analysis methods exhibited significant bias.

Conclusions:

The results of our research give practical guidance in the validity of inferences from outcome-dependent visit processes. When data are subject to outcome dependence, bias is restricted to a subset of the covariates (those with associated random effects in the outcome model). Standard, maximum likelihood-based methods such as mixed-model regression often exhibited little bias, even for parameters with associated random effects. Generalized estimating equations methods, especially those based on an independence working correlation, were more susceptible to bias. The diagnostic methods we developed have high power to detect outcome-dependent visit processes.

Limitations:

While our work developed extensive theory to assess the effects of outcome-dependent visits, many of our results were based on simulation studies in which we introduced known degrees of bias. As with any simulation-based approach, the results cannot be known to apply in full generality. However, the breadth of our simulations, the approximate calculations, and sensitivity analyses all support the primary conclusions.

Background

Information collected in the course of regular health care is increasingly being used for clinical research. PCORI funded 11 clinical data research networks with the following goal:

integrate data from … networks that originate in health care systems such as hospitals, health plans, or practice-based networks and securely collect “real-time,” “real-world” health information during the routine course of patient care [to improve the] capacity to support large-scale comparative effectiveness trials, as well as observational studies of multiple research questions, including prevention and treatment.

However, the results of using such data are subject to bias because the presence of a visit (and hence data to analyze) may be related to the outcome being studied.¹ Our focus is different from the missing data literature in 2 respects. First, because data are potentially available for numerous times, the vast majority of the data are missing. Second, our interest is in the situation where all (or almost all) of the data are available on a visit and we do not consider the additional complications of item nonresponse (eg, some predictors missing some of their values).

As an example, in a study of neurological outcomes following microsurgery for a brain arteriovenous malformation,² we used the modified Rankin scale as a measure of neurological disability. A patient may schedule a visit because he or she is noticing neurological deficits and therefore the availability of data may be related to the outcome being measured—an outcome-dependent visit process. A key question is whether this is likely to introduce bias and invalidate scientific conclusions based on common methods of analysis of such data.

Figure 1 shows the modified Rankin score values from 20 patients being followed postsurgery by the Center for Cerebrovascular Research at the University of California, San Francisco. Regularly scheduled visits (based on planned follow-up times) are marked with a green circle and irregular, unplanned visits are marked with a red ×. The presence of irregular visits leaves open the possibility of bias being introduced by an outcome-dependent visit process.

Figure 1

Modified Rankin Scale vs Days Postsurgery.

Since all standard approaches will provide consistent estimation in settings where all visit times are regularly scheduled—ie, non-outcome dependent visit times—our secondary objective is to assess whether the presence of a few regularly scheduled visits that are weakly or not outcome dependent in a data set substantially reduces bias. We call such visits regularly scheduled to distinguish them from visits that can occur not on a regular schedule and may be highly dependent on the outcome, which we call irregularly scheduled visits.

Previous Work to Accommodate Outcome-Dependent Visit Processes

We used a comprehensive array of statistical methods that we identified in a rigorous search of the statistics literature. Investigators have proposed several classes of approaches to analyze longitudinal data subject to outcome-dependent visit times, including inverse weighted marginal models,^3,4 shared random-effects models,^5-10 and generalized linear mixed models that include aspects of the visit history such as subject-specific counts of previous visits as a predictor.¹¹ All the methods purport to accommodate data with informative visit times, but many papers did not consider realistic visit processes.

Tan et al¹² recently published a review of methods to analyze longitudinal data with outcome-dependent visit times; however, they focus on marginal models for continuous responses that consider time effects to be a nuisance while our work considers more general outcome types and focuses on the estimation of within-subject time effects, typically the scientific objective of longitudinal studies. Also, the simulation studies of Tan et al¹² generate visit times as a function of observed data while our simulations generate visit times according to more realistic models that we describe in the Stakeholder Engagement: Structured Clinician-Scientist Interviews section.

We begin by reviewing the approaches and the types of visit processes they are designed to address.

Weighted Marginal Models

In a series of papers, BU̇rŽKOVÁ et al^3,4 developed an approach known as inverse intensity rate ratio-weighted generalized estimating equations (GEEs) that accommodates outcome-dependent visit times in marginal regression models by weighting GEEs inversely to (roughly) the relative probability that individual i has an observation at time t. Specifically, one calculates the inverse weights as

ρ_{i} (t; γ_{0}) = \exp {γ_{0}^{T} W_{i} (t)},

(1)

where the vector W_i(t) includes observed values of covariates that accurately describe the visit process and γ₀ is the vector of regression coefficients obtained from the fit of a Cox proportional hazards model to the visit times. The vector W_i(t) can include observed current and prior values of x, observed prior values of Y, and the observed values of auxiliary variables related to the visit process but not part of the model for E[Y_ij | X_ij]. Essentially, this approach views the visit process as a sampling problem and corrects the estimating function for the parameters of E[Y_ij | X_ij] in the spirit of Horvitz and Thompson¹³ by weighting inversely proportional to the intensity of a visit. BU̇rŽKOVÁ et al^3,4 note that the observation times model must accurately represent the relationship between the observation times and the observed values of W_i(t) to obtain consistent estimators.

Joint Models for Outcomes and Visit Times

Several authors^5-10 have proposed to accommodate outcome-dependent visit times by developing methods that jointly model the response of interest and the visit process using shared random effects. This approach postulates that the mechanism for the dependence of the visit process on the responses is the shared random effects. A different approach hypothesizes that the visit process depends on the outcome process only through the previously observed outcomes.⁵ Under this assumption they show that the joint model likelihood for the observed data separates into 2 components—1 for the longitudinal outcome and 1 for the visit process—and that one can obtain consistent estimates by maximizing only the likelihood for the longitudinal outcome. We note this approach requires that unobserved visits be missing at random¹⁴ and discuss the lack of reasonableness of this assumption in practice in the Stakeholder Engagement: Structured Clinician-Scientist Interviews section. Lipsitz et al⁵ note that, even when the missing at random assumption holds, there is a potential for biased estimation when the covariance structure of the outcome process is misspecified.

Adjusting for Previous Number of Visits

Rather than jointly model both the outcome and visit processes, Sun et al¹¹ proposed a 2-stage approach in which one first models the visit process marginally and then models the outcome process conditional on the visit process. One could model the outcome process conditional on a variety of functions of the visit process, but Sun et al¹¹ suggest that a natural and simple approach would be to include a variable that records the number of visits before the current observation time in the model for the longitudinal response of interest. One possible approach would be to extend standard mixed-effects models by adding the number of prior visits as an additional covariate. Intuitively, one would expect that responses from a participant who made many visits would differ from a participant who made few visits and including the number of prior visits as a covariate could control for these differences.

Previous Work on Diagnostic Methods

As discussed in the “Joint Models for Outcomes and Visit Times” section, joint modeling of the outcome and visit process has been widely suggested. These joint models often contain parameters that quantify the association between the outcome and visit process and could be used to form a test of an outcome-dependent visit process. However, it is often complicated to fit these joint models and they can be highly problem specific. Furthermore, misspecification of the joint model can lead to incorrect inference.¹⁵ There have been some suggestions for diagnostic tests in the related, but much more restrictive, informative cluster size literature.^16,17 However, this literature is meager and, in fact, Nevalainen et al¹⁷ claim that other than their and the Benhin et al¹⁶ proposals there is “no other test for the presence of informative cluster size problems.”

Summary

Disadvantages to all of the above-described approaches are the requirements to correctly specify models for the visit process (which is often of little interest compared with the outcome process) as well as the dependence between the outcome and visit process, which can be quite complicated in the more recently proposed methods. Because of this complication, an analyst may consider fitting a standard mixed-effects model,¹⁸ which makes no attempt to model the visit process. Like the approaches of Lipsitz et al⁵ and others, fitting standard mixed-effects models will provide consistent estimation when the unobserved visits are missing at random.

Our overarching approach is quite different from those above: We focus on the outcome process and ask whether standard statistical methods, such as mixed-model regression, can unbiasedly and efficiently estimate certain parts of the model (under a wide variety of realistic outcome-dependent visit models). We supplement those investigations with diagnostic methods for uncovering outcome-dependent visit processes.

Stakeholder Engagement: Structured Clinician-Scientist Interviews

To better understand the visit process and to gather preliminary data on realistic models for outcome-dependent visit processes, we conducted semistructured interviews of 4 clinician-scientists with whom we have had long-standing collaborations and who oversee clinical databases. We asked them to specify a typical longitudinal study for which they might use their clinical database and the likely processes that determine regular and irregular visits that would generate entries in their databases. The research areas covered a wide span and included the following: (1) studies of patients undergoing brain surgery for a brain aneurysm (a bulging artery susceptible to bursting) and being followed postsurgery, (2) chronic kidney disease patients being monitored before going on dialysis or receiving a kidney transplant, (3) patients receiving bone marrow transplants being followed posttransplant, and (4) “watchful waiting” in monitoring men diagnosed with prostate cancer.

We also asked for data on the visit processes and follow-up on reasons for missed visits or irregular visits for between 50 and 100 of the patients. This worked well as it gave us structured, qualitative data, open-ended responses, and actual sample data. We conducted data analysis of the visit patterns and used the data to design our simulation studies.

The interviews and data analysis consistently identified several features of the visit process:

Many of the visits are unplanned and are dependent on the patient feeling ill or physician concern and this information is not typically available to a data analyst.
Patients commonly miss visits for reasons related to their condition and information on those reasons is not measured or recorded.
Intervisit timing is highly irregular and does not appear to follow any easily specified stochastic process.
Databases often include subclasses with different visit patterns that relate to the response. For example, patients may be on different treatment regimens such as kidney-toxic drugs that require more frequent monitoring, generating greater measurements of the response.

Notable Impacts of Stakeholder Engagement

Engagement with our stakeholders lead directly to a more realistic set of models on which to base both our theoretical work and our simulation studies. Importantly, we came to understand that none of the models currently proposed in the statistical literature to deal with outcome-dependent data were realistic. For example, features 1 and 2 above indicate that the missing at random assumption made by many researchers is unreasonable.

This calls into doubt the appropriateness of many extant statistical methodologies. Thus, our research should be much more directly relevant to the proper analysis of data subject to outcome-dependent visit processes. That, in turn, should lead to more rigorous analysis of research data to better inform health care decisions.

Our analysis of visit pattern data was symbiotic. The analysis for one of our stakeholder's clinical databases made them realize that they needed to be much more rigorous in guaranteeing a regular follow-up schedule that they had not been achieving.

Methods

Research Design

We begin by defining what we mean by an outcome-dependent visit process. Let V denote the visit process data (eg, all the visit times), Y the outcome data, and X the covariate information. Our basic definition of an outcome independent visit process is that the joint distribution of Y and V factors into separate distributions. Using the usual bracket notation for distributions,

\begin{matrix} \begin{matrix} [Y, V | X] & = [V | Y, X] [Y | X] \end{matrix} \\ = [V | X] [Y | X] . \end{matrix}

(2)

That is, Y and V are independent, conditional on X. We define outcome dependence as any situation in which the factorization in equation (2) does not hold.

Note that under this definition of outcome independence, dependence of the visit process on the covariates is allowed in [V | X]. This is logical when using likelihood-based fitting methods for the following reason. Suppose that the parameters governing the distribution [Y | X] are separate from those of [V | X]. Then maximization of the joint likelihood of Y and V—the left-hand side of equation (2)—to find the estimates of the outcome model is equivalent to only maximizing [Y | X]. Therefore, by ignoring a visit process that depends on X but not Y we can, under typical regularity conditions, obtain consistent estimators of the parameters of the Y outcome process.

Visit Process Notation

We assume that the outcome, Y_ij, is potentially available on a fine grid of “times,” ranging from 1 to T (eg, daily) on “subject” i = 1, …, m. With this fixed time grid, we can quantify the availability of data with an indicator for each time and each subject. Let R_ij be a binary indicator with R_ij = 1, indicating that Y_ij is observed and is 0 otherwise. Denote the times at which subject i has observations as t_i1, …, t_i,ni.

Outcome Models and Notation

We now describe our notation and model for the outcome, which is assumed to be a generalized linear mixed model. This is a very common and flexible model for longitudinal data situations. With Y_ij as the measurement on occasion j (where j runs from 1 to n_i) on subject i, our model assumes that observations are conditionally independent given the random effects and that our outcome process follows a generalized linear mixed model with random effects, b_i, following a normal distribution:

\begin{array}{l} Y_{i j} | b_{i} \sim independent f_{Y_{i}} i = 1, \dots, m; j = 1, \dots, n_{i} \\ g (E [Y_{i j} | b_{i}]) ≝ η_{i j} = x_{i j}^{T} β + z_{i j}^{T} b_{i} \\ b_{i} \sim i . i . d . N (0, Σ_{b}) . \end{array}

(3)

In this model, x_ij represents the covariate vector associated with subject i at occasion j, β (which is of primary scientific interest) is the vector of covariate effects, z_ij is the model matrix for the random effects, g(·) is the link function, and η_ij is the conditional linear predictor. Continuing the example from the introduction, Y_ij, would be the modified Rankin score for patient i at time j, and covariates could include factors such as time since surgery, age, sex, and whether the patient had a history of hemorrhage. Because each person has a different disease severity and may recover at a different rate, the random effects might include (correlated) subject-specific intercepts and slopes with time since surgery.

Data Sources and Data Sets

As described in the Stakeholder Engagement: Structured Clinician-Scientist Interviews section, we conducted semistructured interviews of 4 clinician-scientists who oversee clinical databases. We asked them to specify a typical longitudinal study for which they might use their clinical databases and the likely processes that determine regular and irregular visits that would generate entries in their databases. We also asked for data on the visit processes and follow-up on reasons for missed visits or irregular visits for between 50 and 100 of their patients.

Evaluative Framework

Conditional Distribution of Y Conditional on Being Observed

The key theoretical results are based on assessment of the nature and magnitude of bias introduced by the outcome-dependent visit process. The bias is introduced through the process of selecting some data to be observed—ie, R_ij = 1. Therefore, our theoretical investigation starts by working out the conditional distribution of the outcome, in equation (3), conditional on the event R_ij = 1. We worked this out exactly using a log link relationship, detailed below, and derived approximate results under more general conditions.

Simulation Studies for Assessing Bias

To assess the bias introduced by the outcome-dependent visit process, we also conducted extensive simulation studies. We simulated data using random intercept and slope models from the 3 most commonly used outcome distributions in equation (3)—normal, binary, and Poisson—and for a range of degree of outcome dependence from none to strongly dependent so that we could look for a dose response as the outcome dependence strengthened. We allowed for 3 forms of outcome dependence: on lagged values of the outcome, on the conditional linear predictor, η_ij, from equation (3), and on the direct random effects. We tested a range of different cluster and per-cluster sample sizes. We also considered many variations and misspecified models to assess generality of the results. We considered the following variations:

Errors following an autoregressive model
Non-normally distributed and autocorrelated error terms using a hidden Markov model
Non-normally distributed (skewed and heavy-tailed) random-effects distributions
Different variances and correlation among the random effects
Differing percentages of outcome-dependent visits (from 0% to 100%)

We considered a variety of methods for fitting the models to data. We assessed the approaches suggested in the literature and described in the “Previous Work to Accommodate Outcome-Dependent Visit Process” section: inverse weighting methods, joint modeling, and adjusting for visit process history. Because of the poor performance of those methods, our primary focus was on the performance of 3 standard methods of fitting longitudinal data that ignore the outcome-dependent visit process: mixed-model regression fit via maximum likelihood estimation (MLE), GEEs with an independence working correlation (GEE-ind), and GEEs with an exchangeable working correlation (GEE-exch).

Methods for Diagnosing Outcome-Dependent Visit Processes

Another aim of the contract was to develop diagnostic methods to uncover outcome-dependent visit processes. As a starting point, we hypothesized a joint distribution relating intervisit times and the random effects from the outcome process and developed a score test of no outcome dependence. Those theoretical calculations suggested basing tests on the best predicted values of the random effects. That motivated the assessment of the following test statistics in the simulation studies. For more detail and sample code see Appendix D. We considered 2 cases. In the usual case, all visits are used in the diagnostic test and are described as irregular. However, situations may occur in which there are known, regular observations (such as yearly in a cohort study when additional observations are available in between the regularly scheduled ones), in which case we modified the diagnostic tests to accommodate this extra information.

Test association of the intervisit time with best predicted values of the random intercepts and/or slopes or with the best predicted values of the random-effects portion of the linear predictor: We based the test on linear regression of the log transformation (due to skewness) of intervisit times on the specified function of the best predicted values, adjusting for the covariates and using robust standard errors to account for the multiple intervisit times per person. Because the time to the first visit and the time from the last visit are censored, we omitted them. Also, when the simulation included regular visits we omitted them from the calculation.
Test association of the total number of visits for person i with the average value of the best predicted values of the random-effects portion of the linear predictor or the best predicted values of the random intercepts and/or slopes: We based the test on linear regression of number of visits on the specified function of the best predicted values, adjusting for the covariates. We used a log transformation (after adding 0.5) because of the highly skewed distribution. Again, when the simulation included regular visits, we omitted them from the calculation.
Test association of observed visit time j for person i with the time-varying value of the best predicted values of the random-effects portion of the linear predictor or with the best predicted values of the random intercepts and/or slopes: We based the test on Cox regression of times on the specified function of the best predicted values, adjusting for the covariates (excluding time because that is incorporated in the baseline hazard) and using robust standard errors to account for the multiple times per person. We used the time scale rather than the intervisit time to accommodate regular visits as part of the baseline hazard.

Study Outcomes

The primary outcomes in the study were estimates of the bias of the various estimators considered. For the diagnostic methods the primary outcomes were the size and power of the proposed tests.

Analytic and Statistical Approaches

The main methodologies used were theoretical calculations and simulation studies as described above.

Conduct of the Study

A study protocol was not required for this investigation since it was not considered research involving human participants.

Results

More Realistic Outcome-Dependent Visit Models

The results of the clinician interviews indicated that outcome-dependent visit processes are unlikely to be missing at random and that realistic visit processes must allow the probability of a visit to depend on the underlying health or the current or recent outcomes. In addition, allowance needs to be made for functional dependence on covariates (eg, subgroups that are known to have more frequent visits) and on time (to allow for significantly elevated visit probabilities at regularly scheduled visit times). To allow these realistic features, we consider 2 classes of dependence. In the first we allowed almost arbitrary dependence on the structural elements of the outcome process, namely the fixed and random effects. In the second class we allowed dependence on the covariates as well as lagged values of the outcome. The rationale for using lagged values is that a value of Y_ij indicating poor health may trigger the schedule of a visit somewhat later.

In this spirit, our first class of models is of the form

P (Y_{i j} is observed | b_{i}) ≝ P (R_{i j} = 1 | b_{i}) = h^{- 1} (μ_{i j} + γ_{i j}^{T} b_{i}) .

(4)

In this model, γ_ij governs the strength and directionality of the association between the random effects and whether data are observed. The model in equation (4) is a flexible specification; for example, we can allow dependence on where subjects start (ie, through dependence on their random intercept); their trend over time (ie, through dependence on a random slope); or on their true mean value, through the conditional linear predictor of equation (3), η_ij, at the current or previous time points. Importantly, and to allow more realistic models, both μ_ij and γ_ij can depend arbitrarily on either fixed- or time-varying covariates (including time itself to accommodate regular visits). As an illustration of the flexibility of equation (4), suppose that we have a linear mixed model and the visit probability for subject i at time j depends on a linear function of the true mean outcome for that subject at that time:

P (Y_{i j} is observed | b_{i}) = h^{- 1} (α + δ E [Y_{i j} | b_{i}]) .

(5)

Rewriting the argument of h⁻¹(·) in equation (5), we have

\begin{matrix} α + δ E [Y_{i j} | b_{i}] = α + δ (x_{i j}^{T} β + z_{i j}^{T} b_{i}) \\ = (α + δ x_{i j}^{T} β) + δ z_{i j}^{T} b_{i}, \\ ≝ μ_{i j} + γ_{i j}^{T} b_{i} . \end{matrix}

(6)

with $μ_{i j} = α + δ x_{i j}^{T} β$ and $γ_{i j}^{T} = δ z_{i j}^{T}$ . This shows that dependence on the conditional mean of the outcome process is accommodated with the formulation in equation (4).

Because the visit process depends on the unobserved random effects, it is a “missing not at random”¹⁹ process, and even methods such as maximum likelihood-based fits, which are consistent under missing at random assumptions, may be biased.

In our second class of models, instead of allowing dependence on only the fixed and random effects, we allow dependence on the actual outcomes:

P (Y_{i j} is observed | b_{i}) = h^{- 1} (μ_{i j} + γ Y_{i, j - τ}),

(7)

where τ is the degree of lag by which the outcome triggers visits. While these classes of models were motivated by our clinician interviews, they are not novel and have been studied previously.^20,21

Bias Under the Outcome-Dependent Visit Process

Theoretical Calculations

In this section, we describe the bias that can be introduced by an outcome-dependent visit process. We start with a log link model for the relationship in equation (4) for the outcome dependence because that leads to clean theoretical results. With a log link specification, namely,

P (Y_{i j} is observed | b_{i}) = P (R_{i j} = 1 | b_{i}) = exp{μ_{i j} + γ_{i j}^{T} b_{i}},

(8)

it is possible to derive the exact marginal distribution of Y_ij conditional on R_ij = 1 for any generalized linear mixed model given by equation (3). The derivation is given in McCulloch et al²² with the result that the outcome process is the same as equation (3) except that conditioning on R_ij = 1 modifies the mean of the random-effects distribution from 0 to Σ_bγ_ij. This is an interesting “closure” result since the distribution of Y and b are unchanged except for the mean of the random-effects distribution. Conditional on being observed, the marginal distribution of the outcome is given by the following model:

\begin{matrix} g (E [Y_{i j} | b_{i}]) = x_{i j}^{T} β + z_{i j}^{T} Σ_{b} γ_{i j} + z_{i j}^{T} b_{i} \\ b_{i} \sim i . i . d . N (0, Σ_{b}) . \end{matrix}

(9)

In particular, this suggests that only covariate effects associated with random effects in equation (9) will be biased and others may be estimated unbiasedly.

This exact conditional result can be extended via approximation to outcome-dependent visit processes that are more realistic than equation (8), for example a logistic link model. Let h be an arbitrary link function; ie, $h (P {R_{i j} = 1}) = μ_{i j} + γ_{i j}^{T} b_{i}$ . Expanding $H (u) = \exp {h (μ_{i j} + γ_{i j}^{T} u)}$ in a Taylor series about u = 0, the result in equation (9) holds approximately but with $z_{i j}^{T} Σ_{b} γ_{i j}$ replaced by $z_{i j}^{T} Σ_{b} γ_{i j}^{*}$ , with

γ_{i j}^{*} = γ_{i j} / {h^{'} [h^{- 1} (μ_{i j})] h^{- 1} (μ_{i j})} .

Simulations to Assess Bias

As noted above, we conducted a large number of simulations to assess the bias that would be introduced by analyzing data that follows an outcome-dependent visit process but is analyzed by the usual methods (ML, GEE-ind, and GEE-exch). The basic longitudinal model we used in the simulations contained fixed effects representing a group effect, a time effect, and a group by time interaction. This reflects the typical longitudinal analysis in which the scientific goal is to compare the changes over time between 2 (or more) groups. It also contained random intercepts and random slopes with time, so that the covariate effects for the intercept and time effects had associated random effects, but the group and group by time interactions did not have associated random effects.

In a first set of simulations we investigated the performance of a variety of estimators: MLE, maximum likelihood adjusting for the number of previous visits (MLN), GEEs with an independence working correlation (GEE), joint models with shared random effects (JTY), and inverse weighting using just the covariates (BZX) and lagged versions of the outcome (BZY). Table 1 shows the results under moderately strong outcome dependence with an outcome following a linear mixed model and a logit link model for R_ij that generated an average of 6.4 outcome-dependent visits per subject. The methods that purport to deal with outcome dependence (BZX, BZY, JTY, and MLN) do not improve on the methods that ignore the outcome dependence (GEE and MLE). The 2 inverse weighting methods (BZX and BZY) are related to the GEE method and give virtually identical performances, with bias in the intercept (β₀) and time effect (β_t) but not the group or interaction effects. The method that fits using MLE but adjusting for the MLN is biased for all the covariate effects (group, time, and the interaction) and performs poorly. Fitting a JTY model performs about on par with standard mixed-model regression (MLE) with slightly less bias for the intercept, the group effect, and the time effect and slightly higher bias for the interaction effect, which is often the key parameter of interest. Overall, the MLE and JTY methods exhibited considerably lower bias than the other methods. The poorer performance of the GEE method compared with the MLE is likely due to the well-known, poorer performance of the GEE method with missing data, especially data missing at random. Although the data simulated are not missing at random, the mechanism may be similar enough that the MLE performs fairly well. The inverse weighting methods likely do not improve on the GEE method because the inverse weighting necessary to remove the bias would depend on unobserved quantities. This was due to the fact that we simulated data in a more realistic way based on the input from clinicians, especially points 1 and 2 in the Stakeholder Engagement: Structured Clinician-Scientist Interviews section. The MLN method confounds time trends with the cumulative sample size and thus leads to biased estimators.

Table 1

Mean Values of Various Fitting Methods for a Linear Mixed Model With an Outcome-Dependent Visit Process That Depends on the Conditional Mean of the Outcome and an Average of 6.4 Observations per Subject.

Focusing on standard estimation methods and as predicted by the theory described in the above section, none of the simulation results indicated appreciable bias for the covariates that did not have random effects associated with them, namely the group and the group by time effects. Figure 2 is a representative example for a linear mixed model in which the outcome dependence is given by equation (4) with a logit link. Both the intercept (β₀) and the time (β₁) effects show appreciable bias, but until the outcome dependence becomes very strong at δ = 0.75, the bias in the group (β₂) or interaction (β₃) effects is minimal. Similar results held when the outcome process was binary or Poisson. More extensive simulation results for normal and binary outcomes are given in Appendix A.

Figure 2

Simulated Mean Values of the MLE and GEE Independence Regression Coefficient Estimators.

Simulations to Assess CI Coverage

Although it was not in the original scope of the project, we also conducted a limited simulation to assess CI coverage for the linear mixed model. Simulation details are given in Appendix A. As expected, coverage is close to nominal for no outcome dependence (δ = 0) and coverage is poor for parameters that are biased (eg, the constant term) with outcome dependence. Coverage was close to nominal with mild to moderate strengths of outcome dependence (δ = 0.25 and 0.5) for the parameters predicted to have low bias (β₂ and β₃) but was poor for strong dependence, especially for the MLE, consistent with the introduction of bias.

Table 2

Coverage Rates of 95% CIs for GEE and Generalized Linear Mixed-Model Fitting Methods for a Linear Mixed Model With an Outcome-Dependent Visit Process That Depends on the Conditional Mean of the Outcome and an Average of 6.4 Observations per Subject.

Incorporating a Few Non-Outcome Dependent Visits

An important variation we assessed was the inclusion of a few observations that were not strongly outcome dependent. This might mimic the situation in which most participants in a study came back for regular visits (with a few missed) but there were also irregular visits. We found that the inclusion of a few regular visits significantly reduced the bias, especially for MLE methods. Figure 3 shows the bias as a function of the number of regularly scheduled visits. As before, the bias in estimators of the group or interaction effects is minimal. What is new in this situation is the degree to which the bias can be reduced by having a few regular visits. Simulations with different numbers of total visits showed similar patterns. Additional details as well as details of the simulation are given in Appendix C.

Figure 3

Simulated Mean Values of the MLE Regression Coefficient Estimators With Different Numbers of Regular Visits With the Total Average Sample Size Fixed at 5.

Sensitivity Analyses

We derived the results in the previous section under the assumption that the outcome process was correctly specified. In practice this will never exactly be the case, so it is important to understand to what degree the results hold when those assumptions are violated. We considered the variations listed above in the “Simulation studies for Assessing Bias” section, which includes both model misspecification and a wider range of parameter settings. Figure 4 shows the results of one of those for which we varied the distribution of the random effects. We simulated data from 3 different random-effects distributions—a normal distribution and 2 distributions chosen from the Tukey(g, h) family, a skewed distribution (g = 0.446, h = 0.05), and a heavy-tailed distribution (g = 0, h = 0.2). Figure 5 shows the 3 densities and the skewness and kurtosis values, which were, respectively, 0 and 3 for the normal, 1.9 and 12 for the skewed distribution, and 0 and 18 for the heavy-tailed distribution.

Figure 4

Simulated Mean Values of the MLE, GEE(Exch), and GEE(ind) Regression Coefficient Estimators.

Figure 5

Random-Effects Distributions Used in the Simulation.

In this simulation, the outcome dependence followed equation (7), the outcome process was a linear mixed model, the number of clusters was 1000, and the average sample size per cluster was approximately 6. The plot is only for the case in which there is moderately strong outcome dependence. The results mirror the theoretical and simulation results above in that the group (β_g) and interaction (β_i) effects exhibit little bias for any of the fitting methods. The bias of the MLE estimator is present but small, even for the intercept (β₀) and (β_t) effects, but the GEE methods exhibit bias. There were no qualitative differences across the different distributions.

Appendix B gives the details of the simulation methodology as well as the results for the other sensitivity evaluations. For linear mixed models, we assessed the following forms of misspecification: autoregressive errors (when assumed to be independent) and non-normally distributed error terms (and hence outcomes). We also varied the variances and correlation among the random effects and considered binary and Poisson outcomes. Broadly, the ML and GEE (with exchangeable working correlation structure) fitting methods performed similarly and better than GEE with an independence working correlation structure. An exception was for Poisson models in which standard software was unable to obtain convergence for GEE exchangeable fits and the MLE performed much better than GEE with an independence working correlation structure.

Diagnostic Methods for Uncovering Outcome Dependence

As noted above, we derived a score statistic under various outcome-dependence models and an intervisit distribution that was hypothesized to follow a gamma distribution. The results of that derivation indicated that diagnostic test statistics should be based on the best predicted values of the random effects. Figure 6 shows the power of the diagnostic tests in a linear mixed model using an outcome-dependent visit process with the dependence on the lagged value of Y when there are all irregular or a mix of regular and irregular visits. In the case of all irregular visits, the tests based on the number of observations for each participant (long dashed lines) were the most powerful with the Cox model (solid lines) and intervisit time-based tests (short dashed lines) performing less well. The tests based on the random-effects portion of the linear predictor (marked with an ×) performed better than the tests based on the best predicted values of the random effects. This is to be expected since the random-effects portion of the linear predictor will be more highly associated with the lagged value of the outcome than the random effects themselves. In the case in which there were a mix of regular and irregular visits, the Cox model approach (which explicitly incorporates both types of visits) slightly outperformed the test based on the number of visits and both of them had significantly higher power than the intervisit process tests. Bias for estimators of the regression coefficients using MLE fits for linear mixed models were minimal in the ranges of outcome dependence that were simulated. However, in some cases the bias for GEE independence was larger. Appendix D gives more details on how we conducted the simulation and additional simulation results as well as tables of the results instead of graphs.

Figure 6

Power of Various Tests for an Outcome-Dependent Visit Process When the Outcome Follows a Linear Mixed Model with m = 100 Subjects and an Average Sample Size of 5.

Discussion

Study Results in Context

Simulation studies showed, using realistic visit processes described by our clinician stakeholders, that techniques developed to address outcome dependence under less realistic visit processes did not perform well. We then developed theory and simulation results for the bias of using standard longitudinal data analysis methods when the data are generated under a generalized linear mixed model but subject to an outcome-dependent visit process. Generally, the theory and simulation studies indicate that estimators of parameters associated with the random effects will be biased (sometimes badly so for GEE independence) but that those not associated with the random effects will be estimated with little or no bias. Furthermore, the inclusion of a small number of visits that are not highly outcome dependent significantly reduced the bias for the covariates associated with random effects. This allows increased precision compared with using only the non-outcome dependent visits.

We also developed and evaluated simple tests for the presence of an outcome-dependent visit process. We based these on the association of the number of visits with functions of the best predicted values. They controlled the size and were either the most powerful test or nearly the most powerful test across all the simulated scenarios. They detected associations with high power before appreciable bias was evidenced in the estimated regression coefficients using MLE fits. These straightforward tests can be simply applied in practice to warn of situations in which care must be taken due to outcome-dependent visit processes that, if severe enough, can bias estimated regression coefficients.

Uptake of study results

The results of this work should immediately inform research using electronic health records and data collected using other outcome-dependent visit processes in the following ways:

Covariate effects associated with random effects in the model should be interpreted with caution as they may be biased.
Covariate effects not associated with random effects in the model should be less subject to bias.
Use of maximum likelihood-based methods are encouraged as they are less susceptible to bias.
Simple diagnostic methods should be employed to detect outcome-dependent visit processes.

Uptake may be limited because it takes time for results to propagate from the statistical to the research literature and because more applied examples of the potential bias will need to be published in subject matter literature.

Study Limitations

As this is primarily a theory and simulation-based investigation, it does not have the usual limitations of a human participant-based study. Nevertheless, we were strongly influenced by 2 features: the semistructured interviews described in the Stakeholder Engagement: Structured Clinician-Scientist Interviews section and the resulting outcome-dependent visit process given by equations (4) and (7). The semistructured interviews were extremely consistent in the information generated, and they also reflect our experience with hundreds of other investigations in medical research. That said, it is certainly possible that other research studies could have different drivers of the number or timing of data availability and our results might not apply to those situations. As we have argued, equations (4) and (7) capture a wide variety of outcome-dependent visit processes but certainly not all. A key class of outcome models we did not consider are time-to-event models such as the Cox proportional hazards model. Similarly, if other models held for a particular research investigation our results might not apply.

Future Research

We are currently working on several projects—the results of which will be published in the applied literature—demonstrating the bias that can be introduced with outcome-dependent visit processes. We are also working on other ways, such as the preliminary investigations reported in the “Simulations to Assess CI Coverage” section, to mitigate the bias. Additional beneficial work will include assessment of the robustness of the conclusions reported here when not all of the assumptions of the statistical models are met (as is invariably true in practice).

Conclusions

The results herein indicate that analysis of data that are driven by patient characteristics and may be related to the outcomes being measured (which we call an outcome-dependent visit process) should be undertaken with care. Methods purported to work with outcome-dependent data did not perform well in our simulations. Most of our focus was on the performance of standard methods of analysis (mixed-model regression and GEEs) for longitudinal data. Investigation of the random-effects structure can provide guidance as to which parameter estimates may be biased (those for covariates associated with the random effects) and therefore interpreted with caution. On the positive side, parameter estimates not associated with the random effects are not likely to be biased due to the informative visit process and MLE estimation methods exhibited little bias for any of the parameters in many scenarios.

References

1.: PCORI. Patient-Centered Outcomes Research Institute Cooperative Agreement Funding Announcement: Improving Infrastructure for Conducting Patient-Centered Outcomes Research. Published 2014. https://www.pcori.org/assets/PCORI-CDRN-Funding-Announcement-042313.pdf
2.: Lawton MT, Kim H, McCulloch CE, Mikhak B, Young WL. A supplementary grading scale for selecting patients with brain arteriovenous malformations for surgery. Neurosurgery. 2010;66(4):702-713. [PMC free article: PMC2847513] [PubMed: 20190666]
3.: BU̇rŽKOVÁ P, Lumley T. Longitudinal data analysis for generalized linear models with follow-up dependent on outcome-related variables. Can J Stat. 2007;35(4):485-500.
4.: BU̇rŽKOVÁ P, Brown ER, John-Stewart GC. Longitudinal data analysis for generalized linear models under participant-driven informative follow-up: an application in maternal health epidemiology. Am J Epidemiol. 2009;171(2):189-197. [PMC free article: PMC2878101] [PubMed: 20007201]
5.: Lipsitz SR, Fitzmaurice GM, Ibrahim JG, Gelber R, Lipshultz S. Parameter estimation in longitudinal studies with outcome-dependent follow-up. Biometrics. 2002;58(3):621-630. [PubMed: 12229997]
6.: Fitzmaurice GM, Lipsitz SR, Ibrahim JG, Gelber R, Lipshultz, S. Estimation in regression models for longitudinal binary data with outcome-dependent follow-up. Biostatistics. 2006;7(3):469-485. [PubMed: 16428260]
7.: Ryu D, Sinha D, Mallick B, Lipsitz SR, Lipshultz SE. Longitudinal studies with outcome-dependent follow-up. J Am Stat Assoc. 2007;102:952-961. [PMC free article: PMC2288578] [PubMed: 18392118]
8.: Liu L, Huang X, O'Quigley J. Analysis of longitudinal data in the presence of informative observation times and a dependent terminal event, with application to medical cost data. Biometrics. 2008;64(3):950-958. [PubMed: 18162110]
9.: Liang Y, Lu W, Ying Z. Joint modeling and analysis of longitudinal data with informative observation times. Biometrics. 2009;65(2):377-384. [PubMed: 18759841]
10.: Zhou J, Zhao X, Sun L. A new inference procedure for joint models of longitudinal data with informative observation and censoring times. Stat Sin. 2013;23(2):571-593.
11.: Sun J, Park D-H, Sun L, Zhao X. Semiparametric regression analysis of longitudinal data with informative observation times. J Am Stat Assoc. 2005;100(471):882-889.
12.: Tan KS, French B, Troxel A. Regression modeling of longitudinal data with outcome-dependent observation times: extensions and comparative evaluation. Stat Med. 2014;33(27):4770-4789. [PubMed: 25052289]
13.: Horvitz DG, Thompson DJ. A generalization of sampling without replacement from a finite universe. J Am Stat Assoc. 1952;47(260):663-685.
14.: Little RJA, Rubin DB. Statistical Analysis With Missing Data. 2nd ed. Wiley; 2002.
15.: Chen Z, Zhang B, Albert PS. A joint modeling approach to data with informative cluster size: robustness to the cluster size model. Stat Med. 2011;30(15):1825-1836. [PMC free article: PMC3115426] [PubMed: 21495060]
16.: Benhin E, Rao JNK, Scott AJ. Mean estimating equation approach to analysing cluster-correlated data with nonignorable cluster sizes. Biometrika. 2005;92(2):435-450.
17.: Nevalainen J, Oja H, Datta S. Tests for informative cluster size using a novel balanced bootstrap scheme. Stat Med. 2017;36(16):2630-2640. [PMC free article: PMC5461221] [PubMed: 28324913]
18.: McCulloch CE, Searle SR, Neuhaus M. Generalized, Linear and Mixed Models. 2nd ed. Wiley; 2008.
19.: Fitzmaurice GM, Laird NM, Shneyer L. An alternative parameterization of the general linear mixture model for longitudinal data with non-ignorable drop-outs. Stat Med. 2001;20(7):1009-1021. [PubMed: 11276032]
20.: Wulfsohn MS, Tsiatis AT. A joint model for survival and longitudinal data measured with error. Biometrics. 1997;53(1):330-339. [PubMed: 9147598]
21.: Diggle P, Kenward MG. Informative drop-out in longitudinal data analysis. J R Stat Soc Ser C Appl Stat. 1994:43(1):49-93.
22.: McCulloch CE, Neuhaus JM, Olin RL. Biased and unbiased estimation in longitudinal studies with informative visit processes. Biometrics. 2016: 72(4):1315-1324. [PMC free article: PMC5026863] [PubMed: 26990830]

Acknowledgment

Research reported in this report was [partially] funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (#ME-1306-01466). Further information available at: https://www.pcori.org/research-results/2013/statistical-methods-reducing-bias-comparative-effectiveness-research-when

Appendices

Appendix A.

Detailed Simulations Results Reported in the Supplementary Material for the Biometrics Publication (PDF, 183K)

Table 1. Outcome model: linear mixed model (PDF, 99K)

Informative visit model: log(P(R_it = 1)) or logit(P(R_it = 1)) = −5 + δE[Y|b], Fitting method: maximum likelihood Random effects: corr(b_0i, b_1i)=0

Table 2. Outcome model: linear mixed model (PDF, 85K)

Informative visit model: log(P(R_it = 1)) or logit(P(R_it = 1)) = −5 + δE[Y|b], Fitting method: GEE (independence working correlation) Random effects: corr(b_0i, b_1i)=0

Table 3. Outcome model: linear mixed model (PDF, 79K)

Informative visit model: log(P(R_it = 1)) or logit(P(R_it = 1)) = −5 + δE[Y|b], Fitting method: maximum likelihood, no quadratic term Random effects: corr(b_0i, b_1i)=0

Table 4. Outcome model: linear mixed model (PDF, 79K)

Informative visit model: log(P(R_it = 1)) or logit(P(R_it = 1)) = −5 + δE[Y|b], Fitting method: GEE (independence working correlation), no quadratic term Random effects: corr(b_0i, b_1i)=0

Table 5. Outcome model: linear mixed model (PDF, 79K)

Informative visit model: log(P(R_it = 1)) or logit(P(R_it = 1)) = −5 + δE[Y|b], Fitting method: maximum likelihood Random effects: corr(b_0i, b_1i)=0.5

Table 6. Outcome model: linear mixed model (PDF, 79K)

Informative visit model: log(P(R_it = 1)) or logit(P(R_it = 1)) = −5 + δE[Y|b], Fitting method: GEE (independence working correlation) Random effects: corr(b_0i, b_1i)=0.5

Table 7. Outcome model: linear mixed model (PDF, 100K)

Informative visit model: log(P(R_it = 1)) or logit(P(R_it = 1)) = −5 + γ₀b₀ + γ₁b₁, Fitting method: maximum likelihood Random effects: corr(b_0i, b_1i)=0

Table 8. Outcome model: linear mixed model (PDF, 79K)

Informative visit model: log(P(R_it = 1)) or logit(P(R_it = 1)) = −5 + γ₀b₀ + γ₁b₁, Fitting method: GEE (independence working correlation) Random effects: corr(b_0i, b_1i)=0

Table 9. Outcome model: linear mixed model (PDF, 79K)

Informative visit model: log(P(R_it = 1)) or logit(P(R_it = 1)) = −5 + γ₀b₀ + γ₁b₁, Fitting method: maximum likelihood Random effects: corr(b_0i, b_1i)=0.5

Table 10. Outcome model: linear mixed model (PDF, 79K)

Informative visit model: log(P(R_it = 1)) or logit(P(R_it = 1)) = −5 + γ₀b₀ + γ₁b₁, Fitting method: GEE (independence working correlation) Random effects: corr(b_0i, b_1i)=0.5

Table 11. Outcome model: logistic mixed model (PDF, 99K)

Informative visit model: logit(P(R_it = 1)) = −1 + δE[Y|b], Fitting method: maximum likelihood or GEE (independence working correlation) Random effects: corr(b_0i, b_1i)=0

Table 12. Outcome model: logistic mixed model (PDF, 79K)

Informative visit model: logit(P(R_it = 1)) = −1 + δE[Y|b], Fitting method: maximum likelihood or GEE (independence working correlation) Random effects: corr(b_0i, b_1i)=0.5

Appendix B.

Sensitivity Analyses (PDF, 329K)

Figure 1. Simulated mean values of the maximum likelihood (MLE), GEE-exchangeable and GEE-independence regression coefficient estimators (PDF, 154K)

as a function of the autocorrelation in the errors. Simulated under a lag Y informative visit process with a logit link, i.e., logit(P(R_it = 1)) = −5 + 0.654Y_i,j−5, and linear mixed outcome model with random intercepts and slopes

Figure 2. Simulated mean values of the maximum likelihood (MLE), GEE-exchangeable and GEE-independence regression coefficient estimators (PDF, 154K)

as a function of the bimodality of the error distribution. Simulated under a lag Y informative visit process with a logit link, i.e., logit(P(R_it = 1)) = −5 + 0.654Y_i,j−5, and linear mixed outcome model with random intercepts and slopes

Figure 3. Simulated mean values of the maximum likelihood (MLE), GEE-exchangeable and GEE-independence regression coefficient estimators (PDF, 137K)

as a function of the bimodality of the error distribution. Simulated under a conditional mean informative visit process with a logit link, i.e., logit(P(R_it = 1)) = −5 + 0.654E[Y | b], and linear mixed outcome model with random intercepts and slopes

Figure 4. Simulated mean values of the maximum likelihood (MLE), GEE-exchangeable and GEE-independence regression coefficient estimators (PDF, 154K)

as a function of the random effects variance. Simulated under a lag Y informative visit process with a logit link, i.e., logit(P(R_it = 1)) = −5 + 0.654Y_i,j−5, and linear mixed outcome model with random intercepts and slopes

Figure 5. Simulated mean values of the maximum likelihood (MLE), GEE-exchangeable and GEE-independence regression coefficient estimators (PDF, 153K)

as a function the outcome distribution. Simulated under a non-informative visit process with a logit link, i.e., logit(P(R_it = 1)) = −5 + 0Y_i,j−5, and generalized linear mixed outcome model with random intercepts and slopes

Figure 6. Simulated mean values of the maximum likelihood (MLE), GEE-exchangeable and GEE-independence regression coefficient estimators (PDF, 154K)

as a function the outcome distribution. Simulated under a lag Y informative visit process with a logit link, i.e., logit(P(R_it = 1)) = −5 + 0.654Y_i,j−5, and generalized linear mixed outcome model with random intercepts and slopes

Appendix C.

Varying Numbers of Regular Visits (PDF, 208K)

Table 13. Means of estimated intercepts, β₀, from six approaches fitted to data (PDF, 90K)

simulated from linear mixed effects models with informative visit processes dependent on conditional linear predictors of the response for four strengths of informativeness, γ_Y and five different visit patterns. True β₀ = 0

Table 14. Means of estimated group effects, β_g, from six approaches fitted to data (PDF, 61K)

Table 15. Means of estimated time effects, β_t, from six approaches fitted to data (PDF, 61K)

Table 16. Means of estimated interaction effects, β_I, from six approaches fitted to data (PDF, 61K)

Table 17. Means of estimated intercepts, β₀, from six approaches fitted to data (PDF, 69K)

simulated from linear mixed effects models with informative visit processes dependent on a lag one response for four strengths of informativeness, γ_Y and five different visit patterns. True β₀ = 0

Table 18. Means of estimated group effects, β_g, from six approaches fitted to data (PDF, 60K)

Table 19. Means of estimated time effects, β_t, from six approaches fitted to data (PDF, 61K)

Table 20. Means of estimated interaction effects, β_I, from six approaches fitted to data (PDF, 61K)

Table 21. Means of estimated intercepts, β₀, from six approaches fitted to data (PDF, 79K)

simulated from mixed effects logistic models with informative visit processes dependent on conditional linear predictors of the response for four strengths of informativeness, γ_Y and five different visit patterns. True β₀ = −1.0

Table 22. Means of estimated group effects, β_g, from six approaches fitted to data (PDF, 61K)

Table 23. Means of estimated time effects, β_t, from six approaches fitted to data (PDF, 61K)

Table 24. Means of estimated interaction effects, β_I, from six approaches fitted to data (PDF, 61K)

Table 25. Means of estimated intercepts, β₀, from six approaches fitted to data (PDF, 79K)

simulated from mixed effects logistic models with informative visit processes dependent on a lag one response for four strengths of informativeness, γ_Y and five different visit patterns. True β₀ = −1.0

Table 26. Means of estimated group effects, β_g, from six approaches fitted to data (PDF, 60K)

Table 27. Means of estimated time effects, β_t, from six approaches fitted to data (PDF, 61K)

Table 28. Means of estimated interaction effects, β_I, from six approaches fitted to data (PDF, 61K)

Appendix D.

Diagnostic Test Simulation (PDF, 229K)

Table 29. Power of various tests for an outcome dependent visit process when the outcome follows a linear mixed model with m = 100 subjects and an average sample size of 5 (PDF, 78K)

Outcome dependence is on a lagged value of the outcome. Results are presented for the case of all irregular visits (top panel) or a mix of regular and irregular visits (bottom panel). The tests look for dependence of the actual visit times (Cox), intervisit times (IVT) or the number of visits (n_i or $n_{i j}^{*}$ ) on the best predicted values of the random intercept and slope (b0b1) or the random effects portion of the linear predictor (zb)

Table 30. Mean values of parameter estimates from a maximum likelihood linear mixed model fit that ignores outcome dependence when the outcome follows a linear mixed model with m = 100 subjects and an average sample size of 5 (PDF, 79K)

Outcome dependence is on a lagged value of the outcome. Results are presented for the case of all irregular visits (top) or a mix of regular and irregular visits (bottom) and a range of outcome dependence, δ_Y

Table 31. Mean values of parameter estimates from a GEE independence linear model fit that ignores outcome dependence when the outcome follows a linear mixed model with m = 100 subjects and an average sample size of 5 (PDF, 79K)

Table 32. Power of various tests for an outcome dependent visit process when the outcome follows a linear mixed model with m = 100 subjects and an average sample size of 9 (PDF, 78K)

Table 33. Mean values of parameter estimates from a maximum likelihood linear mixed model fit that ignores outcome dependence when the outcome follows a linear mixed model with m = 100 subjects and an average sample size of 9 (PDF, 79K)

Table 34. Mean values of parameter estimates from a GEE independence linear model fit that ignores outcome dependence when the outcome follows a linear mixed model with m = 100 subjects and an average sample size of 9 (PDF, 79K)

Table 35. Power of various tests for an outcome dependent visit process when the outcome follows a linear mixed model with m = 100 subjects and an average sample size of 5 (PDF, 78K)

Outcome dependence is on the conditional linear predictor. Results are presented for the case of all irregular visits (top panel) or a mix of regular and irregular visits (bottom panel). The tests look for dependence of the actual visit times (Cox), intervisit times (IVT) or the number of visits (n_i or $n_{i j}^{*}$ ) on the best predicted values of the random intercept and slope (b0b1) or the random effects portion of the linear predictor (zb)

Table 36. Mean values of parameter estimates from a maximum likelihood linear mixed model fit that ignores outcome dependence when the outcome follows a linear mixed model with m = 100 subjects and an average sample size of 5 (PDF, 79K)

Outcome dependence is on the conditional linear predictor of the outcome. Results are presented for the case of all irregular visits (top) or a mix of regular and irregular visits (bottom) and a range of outcome dependence, δ_Y

Table 37. Mean values of parameter estimates from a GEE independence linear model fit that ignores outcome dependence when the outcome follows a linear mixed model with m = 100 subjects and an average sample size of 5 (PDF, 79K)

Table 38. Power of various tests for an outcome dependent visit process when the outcome follows a linear mixed model with m = 100 subjects and an average sample size of 5 (PDF, 78K)

Outcome dependence is on the random intercept. Results are presented for the case of all irregular visits (top panel) or a mix of regular and irregular visits (bottom panel). The tests look for dependence of the actual visit times (Cox), intervisit times (IVT) or the number of visits (n_i or $n_{i j}^{*}$ ) on the best predicted values of the random intercept and slope (b0b1) or the random effects portion of the linear predictor (zb)

Table 39. Mean values of parameter estimates from a maximum likelihood linear mixed model fit that ignores outcome dependence when the outcome follows a linear mixed model with m = 100 subjects and an average sample size of 5 (PDF, 70K)

Outcome dependence is on the random intercept. Results are presented for the case of all irregular visits (top) or a mix of regular and irregular visits (bottom) and a range of outcome dependence, δ_Y

Table 40. Mean values of parameter estimates from a GEE independence linear model fit that ignores outcome dependence when the outcome follows a linear mixed model with m = 100 subjects and an average sample size of 5 (PDF, 79K)

Table 41. Power of various tests for an outcome dependent visit process when the outcome follows a logistic mixed model with m = 200 subjects and an average sample size of 9 (PDF, 78K)

Table 42. Median values of parameter estimates from a maximum likelihood logistic mixed model fit that ignores outcome dependence when the outcome follows a logistic mixed model with m = 200 subjects and an average sample size of 9 (PDF, 79K)

Table 43. Median values of parameter estimates from a GEE independence logistic model fit that ignores outcome dependence when the outcome follows a logistic mixed model with m = 200 subjects and an average sample size of 9 (PDF, 79K)

Table 44. Power of various tests for an outcome dependent visit process when the outcome follows a logistic mixed model with m = 200 subjects and an average sample size of 9 (PDF, 78K)

Table 45. Median values of parameter estimates from a maximum likelihood logistic mixed model fit that ignores outcome dependence when the outcome follows a logistic mixed model with m = 200 subjects and an average sample size of 9 (PDF, 79K)

Outcome dependence is on the random intercept of the outcome. Results are presented for the case of all irregular visits (top) or a mix of regular and irregular visits (bottom) and a range of outcome dependence, δ_Y

Table 46. Median values of parameter estimates from a GEE independence logistic model fit that ignores outcome dependence when the outcome follows a logistic mixed model with m = 200 subjects and an average sample size of 9 (PDF, 79K)

Original Project Title: Methods for Analysis and Interpretation of Data Subject to Informative Visit Times

PCORI ID: ME-1306-01466

Suggested citation:

McCulloch CE, Neuhaus JM. (2019). Statistical Methods for Reducing Bias in Comparative Effectiveness Research When Using Patient Data from Doctor Visits. Patient-Centered Outcomes Research Institute (PCORI). https://doi.org/10.25302/6.2019.ME.130601466

Disclaimer

The [views, statements, opinions] presented in this report are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee.

This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License which permits noncommercial use and distribution provided the original author(s) and source are credited. (See https://creativecommons.org/licenses/by-nc-nd/4.0/

Bookshelf ID: NBK599350PMID: 38232205DOI: 10.25302/6.2019.ME.130601466

PubReader
Print View
Cite this Page
McCulloch CE, Neuhaus JM. Statistical Methods for Reducing Bias in Comparative Effectiveness Research When Using Patient Data from Doctor Visits [Internet]. Washington (DC): Patient-Centered Outcomes Research Institute (PCORI); 2019 Jun. doi: 10.25302/6.2019.ME.130601466
PDF version of this title (1.4M)

In this Page

Background
Stakeholder Engagement: Structured Clinician-Scientist Interviews
Methods
Results
Discussion
Conclusions
References
Acknowledgment
Appendices

Other titles in this collection

PCORI Final Research Reports

Related information

PMC
PubMed Central citations
PubMed
Links to PubMed

Recent Activity

Clear Turn Off Turn On

Statistical Methods for Reducing Bias in Comparative Effectiveness Research When...
Statistical Methods for Reducing Bias in Comparative Effectiveness Research When Using Patient Data from Doctor Visits

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Statistical Methods for Reducing Bias in Comparative Effectiveness Research When Using Patient Data from Doctor Visits

Authors

Affiliations

Structured Abstract

Background:

Objectives:

Methods:

Results:

Conclusions:

Limitations:

Background

Previous Work to Accommodate Outcome-Dependent Visit Processes

Weighted Marginal Models

Joint Models for Outcomes and Visit Times

Adjusting for Previous Number of Visits

Previous Work on Diagnostic Methods

Summary

Stakeholder Engagement: Structured Clinician-Scientist Interviews

Notable Impacts of Stakeholder Engagement

Methods

Research Design

Visit Process Notation

Outcome Models and Notation

Data Sources and Data Sets

Evaluative Framework

Conditional Distribution of Y Conditional on Being Observed

Simulation Studies for Assessing Bias

Methods for Diagnosing Outcome-Dependent Visit Processes

Study Outcomes

Analytic and Statistical Approaches

Conduct of the Study

Results

More Realistic Outcome-Dependent Visit Models

Bias Under the Outcome-Dependent Visit Process

Theoretical Calculations

Simulations to Assess Bias

Simulations to Assess CI Coverage

Incorporating a Few Non-Outcome Dependent Visits

Sensitivity Analyses

Diagnostic Methods for Uncovering Outcome Dependence

Discussion

Study Results in Context

Uptake of study results

Study Limitations

Future Research

Conclusions

References

Acknowledgment

Appendices

Appendix A.

Appendix B.

Appendix C.

Appendix D.

Suggested citation:

Disclaimer

Views

In this Page

Other titles in this collection

Related information

Similar articles in PubMed

Recent Activity