An official website of the United States government
NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Susan dosReis, PhD, Gloria Reeves, MD, Beverly Butler, BS, and C. Daniel Mullins, PhD.
Author Information and AffiliationsSurrogate decision makers for children from birth to 21 years with intellectual/cognitive disabilities and mental health comorbidities must make difficult decisions regarding risk and benefit of moderately effective treatments. Methods are needed to determine what outcomes surrogates most value to help them select the best treatment alternative for their child.
The objectives were to (1) elicit surrogate-defined meaningful outcomes for their child with developmental impairment and a coexisting psychiatric disorder, (2) identify the priorities for outcomes for English- and Spanish-speaking surrogates, and (3) apply valid and reliable approaches for generating meaningful evidence to inform surrogate decision-making.
We recruited the 898 surrogates were recruited from racial/ethnic and socioeconomic groups often underrepresented in patient-centered outcomes research. We elicited desired outcomes from 58 surrogates across in-depth interviews and focus groups (aim 1). We pilot tested a paper-and-pencil survey with 38 participants who completed a discrete choice experiment (DCE) and a best-worst scaling (BWS) experiment (aim 2); 802 participants completed the online survey (aim 3). Descriptive and bivariate analyses characterize the sample. Conditional logit generated utility coefficients for the DCE and BWS data. Stratification age, race, and latent class assessed subgroup differences in preferences for desired outcomes.
Participants were mostly the mothers of the child and on average aged 41 to 46 years. Treatment options dealing with the child's behavior, dealing with schools, societal understanding of the child, and access to services were important decision-making attributes. Desired outcomes included school functioning, social relations, future independence, and behavioral functioning. Surrogates' decision-making process spanned several domains. They preferred the child being cared for at home (u = 0.586; 95% CI, 0.53-0.64) and were willing to accept changes in the child's weight (u = 0.151; 95% CI, 0.21-0.09) and mood (u = 0.135; 95% CI, 0.07-0.20) over changes in personality (u = −0.286; 95% CI, −0.35 to −0.23). Spanish-speaking surrogates' decision-making processes involved treatment options that included medication, behavioral therapy, and social skills (u = 0.15; 95% CI, 0.07-0.24) and an interpreter at medical visits (u = 0.23; 95% CI, 0.15-0.31). Desired outcomes of treatment decisions were the child not hurting himself or herself (u = 1.57) or other children (u = 1.11), the child being able to live independently on his or her own as an adult (u = 0.75), and the child being able to manage his or her own health care decisions in the future (u = 0.71).
Surrogate decision makers' preferences for health care treatments and desired outcomes encompass a range of functional outcomes that often are not accounted for in clinical practice. The methodological contribution of this research was the use of continuous stakeholder engagement in linguistically diverse populations that demonstrated feasible and valid approaches to integrating qualitative and quantitative methods for patient-centered outcomes research.
Participants were primarily female mothers of the child (results may not reflect the views of fathers), college educated, married, and in upper-middle-class families. We specifically recruited first-generation Hispanic surrogates in order to include their voice in research.
According to the DSM-5, a developmental disability reflects significant limitations in intellectual and adaptive functioning in conceptual, social, and practical life skills that begin during developmental periods.1 These impairments may prevent youth from ever providing significant feedback on their preferences for treatment options. Thus, these youth may never have the capacity to voice their preferences for health care and must rely on a surrogate to make those decisions on their behalf. A surrogate, defined by the Council on Ethical and Judicial Affairs and adopted by the American Medical Association, is a “a person who is authorized by state statute, case law, or a patient's health care team to make health care decisions on the patient's behalf.” This is unlike a proxy, which is someone the patient appoints when the patient's care decision may be predetermined (eg, Advance Directives).2
Engagement of surrogates in clinical decisions came to the forefront following the passage of the Affordable Care Act and inclusion of patient preferences in decision-making by professional organizations. For one, the American Academy of Pediatrics emphasizes the consideration of family preferences when developing a treatment plan for children with attention-deficit/hyperactivity disorder (ADHD).3 However, patient-centered outcomes research (PCOR) has not considered surrogate-reported preferences for health care decision-making or for health care outcomes for children with developmental and comorbid mental health conditions. Exclusion of surrogate-reported outcomes preferences means PCOR findings may not generalize to patient populations unable to voice their own health care preferences. As a result, patients' key questions are unanswered, such as the following: What can I do to improve the outcomes that are most important for the patient in my care? How can clinicians and the care delivery systems they work in help me make the best decisions about health and health care for my patient? These unanswered questions highlight the need for PCOR that would help surrogates make informed health care decisions for their child.
An estimated 16.8 million people in the United States spend 20 to 30 hours per week in the role of a caregiver (ie, surrogate) for a youth with special needs, mainly a developmental disability (61%), such as autism, a specific learning disability, or a communication disorder; and a mental illness (57%)4 such as ADHD, depression, or anxiety.4,5 Nearly 14% of youth aged 3 to 17 years have a developmental disability,6 of which 10% to 80% have a comorbid mental illness7 and 40% display aggressive behavior8,9 that is chronic and devastating to daily life activities.10
Aggression is increasingly managed using off-label antipsychotic medication (ie, no officially approved indication).9,11-15 As many as 24% of youth with a developmental disability receive psychotropic medications,16 despite moderate to minimal efficacy for aggressive behavior.17-21 Antipsychotic medications have long-term health risks including rapid and significant increased weight gain, waist circumference, and body mass index; lipid abnormalities22; minimal symptom improvement23,24; and a 2- to 3-fold increase in the risk of incident diabetes.25,26 Early childhood exposure to antipsychotics could affect the size and neuronal circuitry of the developing brain.27-29 Surrogates therefore must balance these risks with the benefits of managing symptoms that, if left untreated, could lead to failure in daily life. This responsibility results in having to make tradeoffs in care decisions during that lifespan. Educational accommodations are important for younger children, but as children transition to adulthood management of symptoms that can interfere with independent living, employment, and insurance coverage for health care become critical.
On average, Hispanic youth are diagnosed with autism 2 years later than White children and 6 months later than African American children.30 This may be due to a lower likelihood of having a usual source of care or of seeking care for chronic conditions.31,32 Additionally, lack of Spanish-speaking providers, transportation, and health insurance delay access to care.33,34 Few studies focus on first-generation Hispanic communities' preferences for the care and outcomes of children with these coexisting conditions.
Research with Hispanic communities must address issues with language and communication, shame and stigma, and immigrant status because participation in research could have legal implications on their ability to stay in the United States.35 This may explain why few US investigators have explored the Hispanic community's perceptions of the steps needed to build relationships with key community stakeholders.35 Reducing health care disparities among Hispanic children with special health care needs focuses primarily on providers' cultural sensitivity rather than family engagement in health care decision-making.36 Research has focused on engagement as the mechanism to improve outcomes, but few studies have been designed to directly ask Hispanic families what outcomes are important to them.37 Blanche and colleagues found that challenges in caregiving for a child with autism spectrum disorder revolved around 4 themes: dealing with the diagnosis, dealing with stigma and isolation from family and community, understanding the role of mothers in changing family routines, and utilizing services.38
This study addressed key PCOR questions specific to surrogates of a child with an intellectual/cognitive developmental disability and a coexisting mental illness—What outcomes do surrogates value most when deciding among available treatment options for their child? How can PCOR be enhanced to address the needs of surrogates who provide care for a youth diagnosed with an intellectual/cognitive developmental disability as defined by the DSM-5? The goal was to apply a multimethod approach to elicit the perspectives in 2 populations: English and Spanish. We drew on methods used in medical and health-related research to address 3 specific aims:
We engaged patients and the public from several groups, which by design was a multilevel approach that would bring unique perspectives to the project. Key stakeholders (n = 4) were English- and Spanish-speaking surrogates who were raising or had raised a child with intellectual/cognitive developmental disabilities as well as a mental health condition. The 2 English-speaking key stakeholder surrogates were coinvestigators on a prior National Institute of Mental Health grant and had worked with the principal investigator for 4 years before engaging in the current study. The 2 Spanish-speaking key stakeholder surrogates were new to the project but were seasoned advocates. They were trained by the principal investigator and the project coordinator on the research aspects and, via bidirectional learning, these key stakeholder surrogates educated the research team about the needs of the Hispanic community.
These stakeholders, who were research partners, had a significant role in the full range of research activities, from topic selection, data collection, data analysis, and dissemination. Their mode of engagement was primarily in-person meetings. They brought their own lived experiences to the table and, in this regard, were able to guide the team through each phase of the project (Figure 1). Community leaders (n = 8) represented organizations that included formal advocacy and informal support groups and whose primary contribution was outreach and participant recruitment. The primary mode of engagement was initially in person as well as via email and phone discussions. State administrators (n = 2) had advisory roles, and we shared our progress with them primarily via email.
Key Stakeholders, Community Partners, Community Organizations, and State Organizations in the PIONEER Study.
The researchers and key stakeholders developed a list of organizations and community support groups for outreach and engagement activities. Some organizations were well-known in the Maryland region and others were identified from the respective websites. We scheduled face-to-face meetings for local organizations/groups and telephone calls with organizations in other states, during which we determined the level of engagement each organization/group was able to commit. We identified the person within the organization who worked with surrogates as this was the direct liaison to potential participants.
Key stakeholders assisted with the conceptualization of the study and reviewed the proposal, drafted and revised the English and Spanish surveys, assisted with participant recruitment, ensured the authenticity of engagement efforts, developed and implemented preengagement workshops for the Hispanic community, defined the comparators and outcomes that were most salient to caregivers of youth with impairment in cognitive development and mental health, analyzed qualitative data, contributed to the pilot and field-testing phases, attended monthly progress meetings, helped write the qualitative research findings, and contributed as coauthors on presentations and manuscripts. In addition, the Hispanic key stakeholders helped develop and implement preengagement workshops for the Hispanic community, while leaders primarily assisted with recruitment.
We observed the impact of key stakeholder engagement in the tool development and dissemination of the study findings. This included attending community support group meetings to present the findings and to obtain feedback on the clarity of the information, the representativeness to caregivers' needs, and using this information to help families make informed decisions about his or her child's care. Another important impact was to establish trust with the community and study participants. Finally, the breadth and depth of our outreach efforts are attributed exclusively to the stakeholder engagement.
This mixed methods cross-sectional design had a preengagement period followed by 3 distinct phases (Figure 1). The preengagement set the foundation for our outreach activities with various community organizations. Since the Hispanic community groups were not established formally, our key Hispanic stakeholders advised us to become “known” to the community before we could begin outreach and recruitment. We developed preengagement workshops for the Hispanic community, which are described later. In phase 1, qualitative methods elicited important care decisions and desired outcomes and developed the conceptual frameworks for the next phase. In phase 2, we used qualitative and quantitative methods iteratively to pilot a survey and refine the instrument. In phase 3, we used quantitative methods to analyze the findings from the online survey. The University of Maryland IRB approved the study protocol.
Key stakeholders helped identify the target population of surrogates of youth with cognitive/intellectual developmental disabilities and coexisting mental health conditions. We assembled the samples separately for each study phase.
Eligible surrogates had a child who (1) was aged 26 years or younger; (2) had a DSM-5 cognitive/intellectual developmental diagnosis (eg, learning, communication, autism spectrum disorder); and (3) had a coexisting DSM-5 mental health diagnosis (eg, ADHD, depression, anxiety, mood, conduct, oppositional defiant disorders). The key stakeholders identified and recruited participants and the research team screened all potential participants by phone. Eligible individuals gave written consent in person. To ensure this qualitative research captured the breadth and depth of experiences, we included surrogates of a child up to aged 26 years to elicit the experiences of those currently raising or having raised a child with these comorbid conditions.
Eligible surrogates had a child who (1) was aged 21 years or younger; (2) had a DSM-5 cognitive/intellectual developmental diagnosis (eg, learning, communication, autism spectrum disorder); and (3) had a coexisting DSM-5 mental health diagnosis (eg, ADHD, depression, anxiety, mood, conduct, oppositional defiant disorders). The key stakeholders identified and recruited participants and the research team screened potential participants by phone. Eligible individuals gave written consent in person for the phase 2 pilot survey and consent was obtained as the first part of the online survey in phase 3. Here, we restricted the sample to those aged 21 years because of the phase 1 findings. One of the key care decisions for surrogates was school placement.
Youth with these comorbid conditions are generally in nontraditional educational programs, so it is typical for them to be in school through aged 21 years because of their cognitive deficits. Since school placement would be included in the phases 2 and 3 surveys, it would not be relevant or appropriate to include surrogates of children older than 21 years old as they were no longer dealing with school decisions for their child.
Surrogates were excluded if they (1) were unable to access the internet from home, a library, or other location to complete the web-based survey or (2) had a learning or cognitive disability preventing them from understanding the instructions and giving independent responses.
In phases 1 and 2, our English-speaking key stakeholder surrogates directed low-income and racial minority recruitment by distributing study flyers at support group meetings and reaching out to other community groups in Maryland. For the online survey in phase 3, we outreached to national advocacy organizations via telephone and email to identify organization leaders who would coordinate distribution of the study flyer in their community group. The recruitment targeted individuals for the English version of the survey (described below).
In the preengagement and phases 1 and 2, our Spanish-speaking key stakeholder surrogates directed recruitment of Hispanic surrogates from the surrounding community in Maryland and Washington, DC. This was done via on-site meetings, phone calls, and visits to organizations interested in working with the research team. Participants for phases 1 and 2 were recruited from those who completed the preengagement workshops. In phase 3, we reached out to local and national organizations, support groups, and resource centers that served Hispanic families of children with special needs and for whom Spanish was their first language. The organization leaders distributed the study flyer and referred to the study team those interested in joining the study. This recruitment targeted individuals for the Spanish version of the survey (described below).
The study outcomes were the (1) attributes of a care management plan for the surrogates' child and (2) surrogates' desired outcomes when deciding to use medication for their child.
The key stakeholders and the research team developed the preengagement workshops, the phase 1 interview and focus group field guides, and the phases 2 and 3 survey instruments. To ensure trustworthiness of the information, the key stakeholder surrogates reviewed all data sources before administration.
Four workshops aimed to build relationships with the community; provide a safe space to discuss the needs of families; understand community values, norms, and beliefs; develop culturally sensitive research tools; and assist community partners in building the capacity to educate families of children with special health care needs. A facilitator initiated an icebreaker activity and the key stakeholder surrogate cofacilitated each workshop. Participants learned how research involving caregivers like themselves gives them a voice in the community and how their input contributes to the child's well-being.
Workshop 1—Understanding the Parent explored perceptions of the child's conditions.
Workshop 2—Understanding the Child helped caregivers experience the world of a child through an active learning role-play activity.
Workshop 3—Building the Relationship With My Child used an active learning role-play activity for participants to share their challenges building a relationship with their child.
Workshop 4—Resources for Families emphasized the role of support networks in accessing health care resources for participants' child. It also explained how the Prioritizing Outcomes, Needs, Expectations, and Recovery (PIONEER) study could help other Spanish-speaking caregivers of children with similar conditions.
Nine individual interviews (6 English, 3 Spanish) and 8 focus groups (6 English, 2 Hispanic) conducted with 58 individuals elicited care decisions and outcomes that mattered most to surrogates. Interviews were 1 hour and focus groups were 2 hours: The first hour used a field guide (see Appendix) to focus the dialogue and in the second hour activities assessed the relevance and importance of care decisions and treatment outcomes. We conducted all activities at a community organization facility. Discussions were audio taped and transcribed verbatim. First, we conducted in-depth interviews to obtain the extent of individuals' experiences managing their child's conditions. Next, we conducted focus groups to verify the relevance and importance of the themes that emerged from the interviews and to determine if any key issues were missing. We were able to triangulate the information across the in-depth interviews, focus groups, and our key stakeholder surrogates.
The survey had 4 sections that appeared in the following order: (1) a Likert-type scale assessment of preferences for key care decisions and desired outcomes that we derived from the qualitative interviews and focus groups; (2) a discrete choice conjoint experiment (DCE) of care decision-making priorities; (3) a best-worst scaling (BWS) experiment of outcome preferences; and (4) sociodemographic characteristics of the surrogate and child and the child's current clinical treatment, which was at the end of the survey (see Appendix). We developed 2 surveys—1 from the English-speaking qualitative data and 1 from the Spanish-speaking qualitative data.
BWS, a preferred method increasingly used in health care research,39,40 allowed us to elicit priorities when considering jointly competing concerns associated with desired outcomes for a child. Each question is designed as a choice task profile displaying 6 outcomes and individuals must select the 1 best and the 1 worst outcome from the profile. Since BWS presents the attributes simultaneously (ie, a choice task profile), individuals must make tradeoffs when making a selection; because it reflects a preference for 1 attribute over another, this allowed us to rank attributes in order of priority.
A DCE is also a preferred method used to quantify the relative importance of treatment attributes.41,42 In the DCE, anywhere from 2 to 4 profiles are shown in each choice task question. Three to 7 attributes may be shown in each profile, and individuals select the 1 profile most acceptable to them. Similar to BWS, we evaluated preferences for health care decisions through a structured set of choice task questions in which individuals must make tradeoffs when making a selection.41,43,44
The BWS and DCE offer several advantages over traditional Likert-type response surveys, which elicit preferences considering each attribute individually rather than jointly. This more closely resembles real-life decision-making for which we must consider multiple attributes when making a decision. Further, this allows the estimation of the relative importance of one attribute over another, which cannot be done with traditional Likert-type items. Finally, these methods are less burdensome than are traditional approaches because we can design the survey with fewer questions.
We used a balanced design so attribute and attribute levels for the DCE and BWS were shown an equal number of times to ensure equal likelihood of selection. The DCE was a partial profile design showing 4 out of 7 attribute levels and 3 profiles in each of 12 choice tasks questions. The BWS was a balanced incomplete block design with 16 attributes; 6 attributes were displayed in each of the 16 choice tasks. A paper-and-pencil pilot was administered (n = 38) before the online launch (n = 802). The survey pilot tested the phrasing logic and profile relevance and generated preliminary utility estimates. We modified the survey based on feedback from key stakeholders and the field test.
The in-depth interviews were audio recorded and transcribed verbatim for analysis. Three members of the research team reviewed and, using NVivo version 10 (QSR International; https://www.qsrinternational.com/nvivo/what-is-nvivo), independently coded the transcribed interviews. Following principles of grounded theory and the constant comparative method,45,46 we categorized passages from the transcript using codes that reflected distinct care decisions or desired outcomes. We discussed all coded data and addressed any disagreement until there was 100% consensus, which necessitated in some instances dropping or merging codes. After each code refinement, the research team revisited the transcript using the refined codes to ensure information was not missed. We did this until no new codes were identified, which resulted in a list of candidate attributes.
Following good practices for conjoint analysis and the standards of preferred methods,47-49 we used the qualitative findings to develop our attribute framework. Acknowledging that not all attributes would be equally salient to all individuals, our goal was to cast a wide net across the range of situations that influenced decisions. Feedback from the focus groups narrowed the attribute list to those that would reflect the most important and relevant tradeoffs among alternative care decisions. Focus group discussions were audio recorded, transcribed verbatim, and coded for emerging concepts that supported or expanded the attribute list from the in-depth interview. In addition, focus group participants completed a ranking exercise to select the most important and relevant coded attributes. To identify the variants in care delivery, we selected quotes from the transcribed interviews that represented different aspects of the attribute. Stakeholder advisors analyzed the data alongside the academic researchers and together with the investigative team arrived at a consensus on the levels. Throughout the process we paid careful attention to avoid selecting levels that were too extreme (ie, would never or would always be chosen regardless of the levels of the other attributes shown). In addition, the levels had to represent reasonable and actionable decisions. To ensure the attributes and levels were tradable (ie, a viable alternative), the stakeholder advisors reviewed the proposed attribute levels for relevance to real-world tradeoffs that they and others like them had to make in caring for their child (see exemplary quotes in Figure 2a and participant enrollment for the In-depth Interviews and the focus groups in Figure 2b).40,47,50 Key stakeholder surrogates shared the candidate attribute list with their community constituents as part of the member checking process. The key stakeholder surrogates commented on the accuracy and completeness and determined which attributes should be retained, which were irrelevant, and which should be deleted. DCE and BWS frameworks emerged from this analysis, which resulted in a publication with our key stakeholder surrogates.51 The frameworks were the basis for the quantitative activities in phases 2 and 3.
Exemplary Quotes From the Qualitative Analysis to Inform Attribute Development.
Participant Enrollment for the In-depth Interviews and the Focus Groups.
Participant Enrollment for the Pilot Survey.
Participant Enrollment for the Online Survey.
The pilot survey and online survey underwent the analyses listed below. We conducted the analyses separately for the English and Spanish surveys. We did not make direct comparisons of the regression estimates across the 2 surveys since they were developed from 2 different samples that generated slightly different attributes for the DCE experiment.
Independent variables included the child's age (early vs late adolescence), the surrogate's race (Black, Hispanic, or Asian), and the preference subgroups that emerged from the latent class analysis.
Dependent variables were a binary response selection for (1) care management decision profiles in the DCE and (2) outcomes in the BWS. The response selection is conditioned on the attributes shown in the profile.
Covariates included surrogate age, education, and income, level of support, youth in need of 24-hour care, youth gender, and youth's clinical diagnoses.
Descriptive analyses included bivariate chi-square and 1-way analysis of variance and logistic regression models to characterize the sample overall and by surrogate and youth factors. We examined the DCE and BWS data for potential clustering or item avoidance, suggesting participants were not considering all attributes presented in a choice task profile.
We analyzed the DCE and BWS experiments using a McFadden conditional logit model with effects coding52,53 to estimate the utility score for the overall sample. The utility scores, or mean scores, are the beta coefficient from a conditional logistic regression of the DCE and BWS and from the latent class analysis of the BWS. The utility score is generated for each attribute such that a positive score is interpreted as more preferred and a negative score is least preferred. The absolute value of the score also is important, in that a larger utility score has a strong influence on preferences. In the latent class analysis, a score is generated for each attribute in each latent subgroup. The score interpretation remains the same. We estimated models with robust SEs in Stata 13 (StataCorp LLC; https://www.stata.com).
We estimated preference heterogeneity using Latent GOLD 5.1 (Statistical Innovations; https://www.statisticalinnovations.com/latent-gold-5-1/). We tested models for 1 to 5 classes, selecting the model with the best Akaike Information Criterion and Bayesian Information Criterion model fit statistics that generated theoretically interpretable classes. We examined associations between latent class membership and demographic characteristics using bivariate chi-square tests.
The qualitative portion did not have missing data. For the quantitative part, the research team ensured all paper-and-pencil assessments were completed during the interview, focus group, or pilot survey assessments. Each field in the online survey had to be answered in order to advance to the next question. This avoided missing data or skipped questions.
We learned that (1) to capture the breadth of experiences we needed surrogates currently raising a child as well as those who had raised a child with intellectual/cognitive developmental delays and a coexisting mental health condition, which raised our age criterion to 26; and (2) all youth with this comorbid diagnostic cluster were in nontraditional educational programs through aged 21 years. PCORI and the University of Maryland IRB approved modifications to raise the age criterion to 26 for the interviews/focus groups and age 21 for the pilot/online surveys.
Shortly after we began, we realized that the Hispanic participants did not understand the study purpose and that we needed a separate process for the Spanish survey (it would not be a mere translation of the English-speaking sample). We obtained supplement funds to conduct 4 preengagement workshops in 3 geographic locales in Hispanic communities in Maryland. We successfully recruited the workshop participants into our main study and expanded our Hispanic network nationally.
Since the Hispanic preengagement workshops delayed the qualitative work in the Hispanic community, we completed the qualitative phase with 48 English-speaking participants in order to maintain our PCORI milestones. To reserve some of the sample for the Hispanic qualitative interviews and focus groups, we enrolled 38 English-speaking surrogates (instead of 50) for the survey pilot and 12 reserved for the Hispanic survey development. The survey had to reflect the experiences of the Hispanic community. We obtained PCORI and IRB approval, and as a result, we developed a culturally sensitive survey.
Initially, the eligibility screening was the first component of the online survey. Only individuals who met the criteria would be permitted to complete the survey. This process was not completely reliable, and we obtained PCORI and IRB approval to modify the protocol to a telephone screen for the national recruitment. Individuals who met the screening criteria were emailed a unique study identifier and the survey link.
We recruited the 96 participants for study aim 1 for the in-depth interviews and focus groups and eventually enrolled 58. Overall, 75 individuals (65 English speaking and 10 Spanish speaking) were assessed for eligibility and screened for the in-depth interviews and focus groups (Figure 2b). Of the 75 screened individuals, 48 English-speaking participants were enrolled (17 did not meet the study eligibility criteria). All 10 Spanish-speaking individuals met the study criteria and were enrolled.
Demographic characteristics of the 58 participants for the phase 1 in-depth interviews and focus groups are shown in Table 1. Three-quarters (76%) of the sample were the mothers of the child and 67% were 35 to 54 years old, with the average (SD) age of 46 (12) years. While most (64%) were White, we successfully enrolled Black (16%) and Hispanic (20%) surrogates, most of whom were married at the time of the study. By educational attainment, annual income, and community dwelling, the sample reflects suburban, middle-class families. The children were mostly male and more than half were aged 15 years or older. Common developmental disabilities included intellectual disability (41%) and autism (33%) and common mental health disabilities included ADHD (72%) and anxiety disorder (48%).
Demographic Characteristics of 58 Surrogate Caregiver Participants and Their Child: In-depth Interview and Focus Groups.
The qualitative work in phase 1 identified the core concepts for care decisions and the desired outcomes of treatments that were evaluated in the phase 2 pilot survey. The formative work in phase 1 ensured the concepts were relevant, important, and constituted tradeoffs in decision-making. Some exemplary quotes are shown in Figure 2a.
Three frameworks of attributes, attribute levels, and statements for each attribute level associated with each concept are displayed in Tables 3 to 5. The English and Spanish frameworks in Tables 3 and 4, respectively, were used to develop DCEs that evaluated priorities for care decisions (see Figures 4 and 5 later in this report).
Demographic Characteristics of the 38 Surrogate Caregiver Participants and Their Child: Pilot Surveya.
Conceptual Framework for Care Decisions Among English-Speaking Surrogate Caregivers.
Conceptual Framework for Care Decisions Among Spanish-Speaking Surrogate Caregivers.
Conceptual Framework for Surrogate Desired Outcomes.
Example of a DCE Profile for English-Speaking Surrogate Caregivers.
Example of a DCE Profile for Spanish-Speaking Surrogate Caregivers.
Although the Spanish care decision framework mapped onto similar concepts as the English care decision framework, the wording of the attributes in Table 4 reflected the unique experiences of the first-generation Hispanic families. Concepts identified as important in decision-making were treatment options, dealing with the child's behavior, dealing with schools, societal understanding of the child, and access to services. We used the framework in Table 5 to develop a BWS instrument to evaluate preferences for desired outcomes of treatment (see Figure 6 later in this report). The most desired outcomes domains were school functioning, social relations, future independence, and behavioral functioning. In the qualitative phase with the Spanish-speaking participants, we found this subgroup did not discuss desired outcomes, despite our efforts to probe this during the qualitative interviews. The Hispanic families were recent immigrants (ie, within the past 5-10 years), were new to the support group concept, and were struggling with care options and access to the care they felt their child needed, and this consumed their lives. Thus, we were not able to elicit key outcomes that would be meaningful for these families and determined it would be inappropriate to impose the outcomes we elicited from the English-speaking sample onto our Spanish-speaking sample. Also, given the cognitive burden and limited education for the Hispanic families, we felt it would be too burdensome to include a DCE and a BWS in the Spanish survey. To avoid cognitive burden, the Spanish survey included only the DCE for care preferences.
Example of a BWS Experiment Profile for English-Speaking Surrogate Caregivers.
In the phase 2 pilot survey (Figure 3a; see Figure 3b for the online survey), 56 individuals were assessed for eligibility and screened, of which 4 were not eligible because the child did not have a mental health condition. The pilot survey study enrolled 38 participants; the other 14 individuals were unable to participate due to scheduling conflicts.
The demographic characteristics of the 38 pilot survey participants in phase 2 were very similar to the surrogates that participated in the in-depth interviews and focus groups (Table 2). Characteristics of the children also were similar, with autism (55%) and ADHD (76%) comprising most developmental and mental health conditions, respectively.
We analyzed the DCE and BWS data using conditional logit models with robust variance estimation for utilities and 95% CIs (Tables 6 and 7). The purpose of the analysis was to determine if individuals were making tradeoffs among the different attributes and attribute levels and to test the phrasing of the individual statements to ensure they reflected the key underlying concept. The results show variability within care decision attributes, with the exception of time cost (Table 6). Surrogates preferred to reduce the number of medications the child was taking (u = 0.43; 95% CI, 0.26-0.60) and to maintain guardianship and custody for the care of the child (u = 0.89; 95% CI, 0.70-1.07). They found medication effects on the child's mood more important than their effects on personality or body weight. Surrogates preferred to avoid conflicts by not bringing their child to social events (u = 0.25; 95% CI, 0.05-0.45) and to avoid home schooling (u = −0.52; 95% CI, −0.74 to −0.29).
Utility Estimates for Surrogate Care Preferences From the DCE Pilot Survey.
Utility Estimates for Surrogate Outcomes Preferences From the BWS Experiment Pilot Survey.
Preferences for desired outcomes of treatment decisions are displayed as utility estimates from a conditional logit analysis with robust variance estimates (Table 7). The most important outcome was the child's safety (u = 1.01; 95% CI, 0.78-1.24), followed by getting an individualized education plan (u = 0.77; 95% CI, 0.54-1.01), and being able to stay in school all day without any problems arising (u = 0.53; 95% CI, 0.29-0.77). Surrogates equally preferred 3 attributes, with utilities of 0.40 to 0.41 and 95% CI from 0.16 to 0.66: maintaining safe behavior at home, the child holding a paying job as an adult, and the child living on his or her own as an adult. Least important to surrogates were maintaining family routines (u = −1.17; 95% CI, −1.40 to −0.94), maintaining friendships with peers (u = −0.78; 95% CI, −1.04 to −0.54), and being able to bring the child to social events (u = −0.70; 95% CI, −0.93 to −0.46).
A debriefing discussion session followed each pilot survey in order to determine the need to refine any of the attributes or statements. The discussions with participants indicated the medication use, parental custody, and time cost attributes did need to be refined. We changed medication use to therapeutic effects because participants indicated it was not simply a reduction in medications they wanted but regimens that would reduce the child's symptoms. We changed parental custody to the extent of the surrogate's decision-making authority (eg, full, partial). Finally, we charted the time–cost attribute, which referred to the extent to which surrogates had to rearrange their schedule to attend an appointment with their child's health care provider, to finding the right provider, which reflected having a provider with a flexible schedule.
Various community organizations and support groups nationally recruited the online survey sample. Overall, 892 individuals were assessed for eligibility and screened, of which 70 did not satisfy the mental health condition criterion. Of the 822 eligible participants, 20 did not complete the survey so 802 were enrolled. Following the online survey launch, we inspected the initial responses for integrity of the information. We detected 389 of the completed English surveys were of questionable integrity and 1 individual completed the survey more than 1 time, thereby invalidating the response and rendering the survey unusable. To ensure the integrity of the data, we removed these responses, leaving 413 for the final analytic sample (n = 346 English and n = 67 Spanish). Of note, because outreach was to the community organization, it was not possible to determine how many individuals received the invitation and of those how many were eligible. We acknowledge this as a limitation later in the report.
The demographic characteristics of the sample are shown in Table 8. The characteristics of the participants who completed the English version of the online survey were similar to the characteristics of our local sample for the formative work conducted under aim 1. In most cases, the respondent was the mother of the child and aged 36 to 50 years, with an average (SD) age of 40 to 41 (7-7.9) years. Our English-speaking sample included Black participants in proportions that are similar to or exceed their distribution in the US population. In particular, the Hispanic proportion of the survey sample, which we derived from self-identified Hispanics who completed the English survey and Hispanics who completed the Spanish survey, exceeds the approximately 17% of the US population who are of Hispanic origin. Of the English-speaking survey participants, most were married, college educated, and employed full time, and earned an annual household income of >$50 000. The Hispanic participants who completed the Spanish-survey were less educated and had a lower annual household income.
Demographic Characteristics of the 413 Surrogate Caregiver Participants and Their Child: English and Spanish Online Survey.
For both English- and Spanish-speaking survey participants, children were mostly male and aged 10 or younger, with an average (SD) age of 10 to 11 (~4-5) years. Most children were in elementary school (58%) and a small percentage was not able to attend school or were in a special program. Among survey respondents, 59% of English-speaking participants and 82% of Spanish-speaking participants reported public insurance coverage for their child's medical care. Autism was the most common disorder, particularly among those who completed the Spanish survey (64% vs 38% of English-speaking participants). Anxiety was the most common mental health condition among the English-speaking participants (52%), followed by ADHD (45%). However, ADHD (64%) was the most common mental health condition among Spanish-speaking participants, followed by anxiety (51%). Just less than a third (31%) of English-speaking participants had a child diagnosed with depression and 22% of Spanish-speaking participants had a child diagnosed with conduct disorder. English-speaking families reported their child was aggressive toward property (57%), toward other individuals (53%), and toward self (41%). Most (58%) Spanish-speaking participants did not report their child displayed aggressive behavior toward property, other individuals, or self.
Survey participants described their use of psychosocial and medication treatment (Table 8). English-speaking participants reported family therapy and individual psychotherapy as the most common psychosocial interventions, while the Spanish-speaking participants' most common psychosocial interventions were speech and language therapy (79%), followed by occupational therapy (58%). For current counseling, 54% of English-speaking participants reported family counseling with the child compared with only 6% of Spanish-speaking families. Most Spanish-speaking participants (64%) did not receive counseling. Overall, 86% of participants indicated their child was taking a psychotropic medication. Most children were taking medication for ADHD, depression/mood, and anxiety; fewer reported using medication for sleep or for aggression.
The English care decision framework underwent refinement after the pilot survey. We changed 3 attributes: medication use, parental custody, and time cost. The refinement of the attribute level statements is displayed in Table 9. The debriefing after the pilot survey did not indicate further refinement of the outcomes framework was needed.
Refinement of the Conceptual Framework for Care Decisions Among English-speaking Surrogate Caregivers.
Therefore, the BWS for the outcome domains included in the online survey remained the same: school functioning, social relations, future independence, and behavioral functioning. The wording of the attributes was unchanged from the pilot survey.
We analyzed the DCE and BWS data using conditional logit models with robust variance estimation for utilities and 95% CIs (Tables 10-12). Preferences for care decisions DCE are displayed in Table 10 by attribute level with the utility coefficient, SEs, z statistic, significance, and 95% CIs. Medication that would not reduce the target symptoms was least preferred among therapeutic effects (u = −0.497; 95% CI, −0.57 to −0.43) and there was a strong preference for the child to be cared for at home (u = 0.586; 95% CI, 0.53-0.64). Surrogates preferred to rearrange their schedule (u = 0.233; 95% CI, 0.17-0.29) rather than take time away from daily responsibilities (u = −0.235; 95% CI, −0.17 to −0.30) to bring the child to a care appointment. Regarding the effects of medication, surrogates preferred changes in the child's weight (u = 0.151; 95% CI, 0.21-0.09) and mood (u = 0.135; 95% CI, 0.07-0.20) over changes in the child's personality (u = −0.286; 95% CI, −0.35 to −0.23). The least preferred option for school placement was a regular public school (u = −0.364; 95% CI, −0.43 to −0.30). Care preferences from the online Spanish DCE (Table 11) show the most preferred were treatment options that include medication, behavioral therapy, and social skills (u = 0.15; 95% CI, 0.07-0.24); an interpreter (u = 0.23; 95% CI, 0.15-0.31); a special program in a regular school (u = 0.17; 95% CI, 0.09-0.25); and avoiding talking to other people about their child's problems (u = 0.21; 95% CI, 0.13-0.29).
Utility Estimates for Surrogate Care Preferences From the DCE English Version of the Online Survey (n = 346).
Utility Estimates for Surrogate Care Preferences From the DCE Spanish Version of the Online Survey (n = 67).
Utility Estimates for Surrogate Outcomes Preferences From the BWS Experiment English Version of the Online Survey (n = 346).
The aggregate results of the BWS preferences for desired outcomes are shown in Table 12. Safety and good adult outcomes were the most important desired outcomes preferred in this sample. The top 5 most important outcomes were the child not hurting himself or herself (u = 1.57), the child not hurting other children (u = 1.11), the child living on his or her own as an adult (u = 0.75), the child not hurting siblings at home (u = 0.75), and the child managing health care decisions in the future (u = 0.71). Least important outcomes were related to the child getting good grades in school (u = −1.31), the child having friends who understand (u = −1.25), and the child having a paying job as an adult (u = −0.69). Many of the social outcomes were less important than the other outcomes when considering treatment decisions.
The subgroup analyses of the English-speaking survey participants included an analysis of different results in groups defined by the child's age and self-identified race/ethnicity (Tables 13 and 14, respectively) and a latent class analysis (Table 15). We found no substantial group differences when comparing surrogates of children aged 10 years or younger with surrogates of children older than 10 years (Table 13). Safety and adult outcomes of the child living on his or her own and managing health care decisions were priorities relative to the other outcomes regardless of the child's age. Stratifying the English-speaking participants by self-reported race/ethnicity (Table 14), White and self-identified Hispanic surrogates (who completed the English survey) were most concerned with the child's safety toward self and other children and with the child being able to manage health care decisions as an adult. Black surrogates were most concerned with the child's safety toward self and others.
Utility Estimates for Surrogate Outcomes Preferences From the BWS Experiment English Version of the Online Survey Stratified by the Child's Age (n = 346).
Utility Estimates for Surrogate Outcomes Preferences From the BWS Experiment English Version of the Online Survey Stratified by Self-identified Race/Ethnicity (n = 346).
Latent Class Analysis of Surrogate Outcomes Preferences From the BWS Experiment of the English Version of the Online Survey.
A 4-segment solution best fit the data in the latent class analysis (Table 15; English survey version only). Class 1 represented 37% of the sample and was most concerned with the child's safety toward self and other children and the ability to manage his or her own health care decisions in the future. Class 2 was 25% of the sample and was solely concerned with the child's safe behavior; all other attributes were negative with the exception of the child living on his or her own as an adult and interference with family routines (which was not significant). Class 3, the next largest segment with 22% of the sample, had strong preferences for adult independence in living, working, and managing money. Class 4 was 16% of the sample and placed more importance on the child's educational supports and the child not hurting himself or herself.
A main finding from this study was that surrogates preferred care options that were not traditional medical interventions. For example, surrogates wanted authority to make all decisions about the child's care and they wanted the appropriate school placement, both of which were more important than the effect of medications and finding the right provider. Further, symptom management was not a desired outcome. Rather, surrogates valued the child's safety and future adult outcomes. In health care, the goal is to reduce symptoms, which could be considered an intermediate outcome. However, what really matters to the family and child is the long-term outcome, at least for our English-speaking sample. This implies that adopting patient-driven outcomes, as opposed to symptom reduction or clinical markers of improvement that are typical in most clinical trials, would better assess the comparative effectiveness of care decisions from the patient and surrogate perspective.
Since this was not an intervention study, we could not measure the impact of surrogate preferences on treatment received in practice settings, but there may be some indirect implications that could inform future trials. Evidence suggests that identifying individuals' most preferred care options can inform patients' or clinicians' decisions to adapt health care delivery to individual patient needs.54 Incorporating preferences in treatment planning can help patients and clinicians determine where there is conflict between the demands of the treatment intervention and the specific needs of a community.54 Adapting treatment attributes to the needs and preferences of the individual can improve health care delivery efficiencies, minimize waste, and reduce ineffective use of limited resources.54 When health care decisions are aligned with one's desired outcomes, individuals are more likely to attain their preferred treatment outcomes.55 For example, in the current study this could mean aligning treatments that could help families achieve desired outcomes, such as preventing the institutionalization of children with intellectual/cognitive and mental health comorbidities, promoting independent living in adulthood, and screening for behaviors that could harm the child, siblings, or others. The findings identify the key domains that are most sensitive to care decisions and preferred outcomes, which could inform policies for patient involvement in treatment decisions. At the individual level, a surrogate's preference for treatment and outcomes is likely to influence what may be best for the patient and his or her family. As the current health care climate encourages greater participation and engagement of patients in actively deciding which treatment is best for them, this will require knowledge of preferred care decisions and the anticipated outcomes. Research findings from this study can be used by clinicians to begin conversations to help patients consider the risks and benefits that are most important given their own situation and, by doing so, better facilitate shared decision-making.56
Finally, health care disparities and practice variation are viewed as primarily due to problems with access to care, yet we know that treatment plans are not always aligned with individual preferences.57-59 When considering the differences between the English and Spanish conceptual frameworks, some of the care decision domains could result in disparities that may be incorrectly viewed as access barriers. For example, for English-speaking participants choosing the right provider was based largely on convenience—how surrogates organize their responsibilities to attend appointments for a child. For the Spanish-speaking participants, the right provider was someone who could facilitate referrals, without which it is difficult to access care. Having the right person to help translate during a medical visit was also important to Hispanic families. Thus, the same attribute has different meaning across ethnic groups and could be an important factor in driving health care access and disparities.
In the time since beginning the present study, several systematic reviews of preference elicitation methods have been published.60-62 None refer specifically to children with special intellectual/cognitive or mental health conditions. In addition, the systematic reviews geared toward clinical conditions identified few pertinent articles, often 50 or fewer. Thus, it is difficult to place the current study in the context of existing evidence. In 2 of the systematic reviews, 60,61 the authors conclude that patient views of treatment and outcomes are important. The authors report that moderate improvement in outcomes may be sufficient for some patients60 and clinical guidelines do not incorporate routinely the preferences of patients.61 In concert with the present study findings, we also found that surrogate preferences were not often aligned with the recommendations of a clinical guideline.
A systematic review of the use of preference elicitation methods examined their utility for clinical decision-making.62 Discrete choice experiment methods, like the approach used in the present study, were used most commonly in studies focused on benefit–risk assessment and clinical decision-making. The use of these methods has grown exponentially since 2012.54 This, along with the advantages over other approaches,56 suggests this is an important method for eliciting patient and surrogate preferences that will have a meaningful impact on health care decision-making.
The methods used in our study are being used more extensively in health care research, and the field is continuing to refine the methods for data collection and analysis. This study contributes to the literature in the application of stated preference methods and to the evidence supporting their application across different populations and especially those with multiple morbidities and in linguistically different populations. Health preference research applies to clinical care in 3 key ways: to promote shared decision-making, to facilitate goal setting, and to assist with advanced care planning.55 By eliciting a surrogate's voice in what matters most in the care decisions and outcomes of his or her child, this input can enhance surrogate–provider shared decisions regarding treatment planning. Our findings hopefully contribute to a deeper understanding of the outcomes that matter most and can assist providers and families with goal setting and revisiting whether treatment is meeting the intended goal.
Although there is great potential to implement a decision tool that utilizes stated preference methods like DCE and BWS into typical care settings to aid in clinical decision-making, additional research is needed. Implementing these methods into practice guidelines or other approaches to influence practice will require further study and represents a gap in the field.58 In fact, stated preference methods as decision support tools have not been evaluated in randomized clinical trials.56 This represents a barrier to implementation, and the hope is that studies such as ours motivate investigators to further test the application in health care decision-making.
The goal of this methods development research was to engage a wide-reaching surrogate audience so the findings would be generalizable. We conducted our initial development work with a local sample and we pilot tested the survey locally in the Maryland region. We developed the survey for online administration nationally. We obtained several local and national features' samples and the findings from our pilot and online survey suggest reasonable generalizability. Compared with the US Census population distribution by race and ethnicity, the pilot and online samples slightly overrepresent minorities, in particular Black and Hispanic communities. Our successful engagement of the Hispanic community was because of the supplemental award we received to conduct preengagement activities with Hispanics. Consequently, we were able to conduct subgroup comparisons to identify differences in preferences among minority groups.
In addition, the findings from the pilot survey based on our local sample were comparable to the findings from the online survey based on our national sample. This lends some confidence that our findings were somewhat generalizable, particularly regarding the child's safety being the most important outcome. However, we acknowledge that replication and additional testing are necessary to better assess generalizability and to identify more diverse and heterogeneous preferences.
An important aspect of the findings is the considerations specific to certain subpopulations in our target population of surrogates caring for a child with intellectual/cognitive developmental delay and mental health comorbidities. It will be important to consider aggression and other risk factors for unsafe behavior and how this influences treatment considerations to achieve a desired outcome. Our latent class analysis revealed 4 subgroups, 3 of which prioritized safety of the child. However, the context in which the safety was prioritized differed (ie, managing medical decisions, adult independence, and safety from harm to self or others). This could be related to the developmental stage of the child. For those transitioning to adulthood, the concerns with independence may be most salient in the context of safety. One of the 4 groups did not prioritize the safety attributes but was most concerned with the child's academic accommodations and learning. It could be surmised that the severity of the aggression may underpin decisions about what matters most.
The study findings can be used to tailor health care interventions to individuals' needs as a way to improve health outcomes. For health policy decision makers, this may inform health care resources allocation to keep youth in the community and to promote adult independence. The findings may also help health care providers engage surrogates and their children in shared decision-making related to treatment planning issues that are important to them. It will be critical to evaluate how incorporating patient-driven outcomes in comparative effectiveness research may change future decisions regarding health care interventions.
Considering the application of the findings to patient care, some could be directly applicable and some should be viewed as exploratory and in need of further research. The attribute domains for care decisions that have been fairly consistent across samples, both English and Hispanic, were issues of school accommodations, surrogate involvement in all decisions, provider access, and the tradeoffs with family relations. The importance of other contextual factors affecting the child's care could be integrated into the treatment planning and accommodations arranged to help families resolve issues that can be barriers to care.
The present study has some limitations worthy of mention. The participants were primarily the mother of the child. It is very likely that the views of the fathers could differ from those of mothers. Most of the surrogates were married, and we were unable to capture the shared decision-making of both parents as the primary caretakers of the child. Although there was reasonable minority and ethnic representation, the English-speaking sample was largely college educated, married, upper-middle-class families. It is possible that we failed to include an attribute relevant to surrogate treatment decisions and desired outcomes. However, including all possible attributes in our DCE or BWS renders greater cognitive burden, which would affect the internal validity of the study.54
The target sample was limited to surrogates of children with both an intellectual/cognitive disability and a mental health comorbidity, and we were not able to determine if the preference estimates differed for surrogates of a child with either an intellectual/cognitive disability or a mental health condition but not both. Finally, since we targeted recruitment through support groups, it was not possible to assess all eligible participants and it is likely our findings are subject to selection bias. Similar findings across the different samples in each phase of the study may help mitigate this concern but cannot completely eliminate this possibility. It is unethical to collect information from individuals who do not consent to enroll in the study, and this limits the possibility of comparing those who refuse to join the study with our enrolled participants.
Several recommendations for further research are offered. One recommendation is to determine how stated preferences for health care interventions correlate with actual behavior. Health care decisions are complex and often influenced by others' opinions. Additional research on other circumstances that may influence health care decisions, such as illness severity, surrogate health, and quality of life, is needed.54 We also recommend generating evidence regarding the implementation of health preferences into clinical practice. Researchers have noted the lack of randomized trials to provide rigorous evidence that would demonstrate the benefit on patient outcomes.56 Further research is needed to use the utility scores as preference weights in analytical models to assess the influence this may have on patient outcomes. A third recommendation is to incorporate patient-centered health preferences into clinical guidelines and to determine if this improves the efficiency of our health care system and better allocates limited resources. Finally, the latent subgroup findings should be viewed as exploratory since this is, to our knowledge, the first study to investigate differences in preferences for desired outcomes in this specific target population. Additional work is necessary to support or refute the findings or identify additional groups that were not possible to detect in our sample.
The contribution of this methodological research can be considered on several levels. First, in the field of stated preference research, qualitative research to elicit the attributes then refine the language used in the attribute statements is the basis for methodological rigor in attribute development.48,49 We based our research on this methodological principle and the good practices for conjoint analysis,47 and we advanced the methods using continuous stakeholder engagement in linguistically diverse communities. Our explicit and transparent methods for eliciting meaningful care decisions to develop a framework of key attributes is a contribution to stated preference methods.63 Most of the published research in stated preferences methods indicates simply that qualitative methods were used, but we describe in detail how the attributes were derived and specifically how we used in-depth interviews to elicit the key domains and used focus groups to validate and assess the importance of the domains.
Second, to our knowledge, prior research has not involved the Hispanic community to the same extent in attribute development. For this work, we developed all the instruments in Spanish, collected all the data in Spanish, and analyzed all the data in Spanish; only the final results were translated into English. We demonstrated that stated preference methods are feasible in a linguistically diverse community.
Third, we demonstrated consistency in our findings to support the feasibility of applying valid and reliable methods to assess population-level priorities in care decisions and outcomes. Consistent findings between the early developmental pilot work with a surrogate sample in 1 geographic locale and the later phase administration with a national sample support the external validity of the research. The sample overrepresented Hispanics, which was our intent since this is a group infrequently recruited into research. We relied on community organizations and advocacy groups for participant recruitment because our goal was to reach the voice of surrogates that often are considered hard to reach. While DCE and BWS can be very powerful tools in preference elicitation, they can be burdensome if not designed well. All participants fully completed the survey and we did not detect erratic response patterns, both of which would suggest cognitive burden.
Research reported in this report was [partially] funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (#IHS-1306-01511) Further information available at: https://www.pcori.org/research-results/2013/understanding-caregiver-preferences-treating-children-intellectual-and
Download PDF (2.8M)
dosReis S, Reeves G, Butler B, Mullins CD. (2019). Understanding Caregiver Preferences for Treating Children with Intellectual and Cognitive Disabilities and a Mental Illness. Patient-Centered Outcomes Research Institute (PCORI). https://doi.org/10.25302/10.2019.ME.130601511
The [views, statements, opinions] presented in this report are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee.
This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License which permits noncommercial use and distribution provided the original author(s) and source are credited. (See https://creativecommons.org/licenses/by-nc-nd/4.0/
Your browsing activity is empty.
Activity recording is turned off.
See more...
