U.S. flag

An official website of the United States government

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Cover of Helping Caregivers of Children with Differences/Disorders of Sex Development Decide on Treatment

Helping Caregivers of Children with Differences/Disorders of Sex Development Decide on Treatment

, PhD, , MA, , PhD, , MD, , MS, , MPH, , MD, MPH, , PhD, , PhD, and , MD, PhD.

Author Information and Affiliations

Structured Abstract

Background:

Differences/disorders of sex development (DSD) encompasses congenital conditions in which chromosomal, gonadal, or anatomic sex development is atypical. Clinical management commonly involves irreversible procedures performed in early life where “right” treatment decisions (eg, genital surgery) do not necessarily follow from specific diagnoses.

Objectives:

Create a decision support tool (DST) to promote shared decision-making in DSD; assess in-clinic communication between parents and health care providers related to decision-making; characterize processes used to gather input for DST development; and summarize use.

Methods:

We divided the project into 3 major phases. In phase 1 (DST development), we interviewed key stakeholders (ie, affected children's parents [hereafter referred to as caregivers], health care providers, patient advocacy leaders, and clinical researchers; n = 49) regarding perspectives/experiences related to DSD and decision-making to guide DST development. In phase 2 (examining family-provider communication during clinic visits), member of the research team audio-recorded caregiver-provider conversations during regular DSD clinic appointments before (n = 36 caregivers) and after (n = 25 caregivers) introducing the DST into regular clinic care. In phase 3 (identifying opportunities and barriers to DST implementation), we elicited stakeholder (n = 40) opinions regarding adequacy of DST content, format, and desirability of integration with regular practice. Data comprised written feedback and transcribed interviews and clinic visits.

Results

DST content and format:

Stakeholders identified the following as essential: providing balanced information about treatment options and associated risks and benefits, diagnostic tests (biochemical, genetic, imaging, exploration under anesthesia) and their timing, and how to interpret “incidental” genetic findings and carrier status; interpreting relationships between test results and gender of rearing decisions; eliciting caregiver values and preferences; using personal stories; coaching caregivers in effective communication with the health care team; disclosing conflicts of interest; addressing health literacy; and delivering an internet-based DST.

Clinic communication:

We analyzed clinic visit content using qualitative computer-assisted coding that captured discussion topics and conversational tools. Overall, the most frequent topic was “medical and anatomic facts”; however, compared with all nonpsychologist providers, caregivers spent less time on these facts and more time on “psychosocial implications of the DSD and its management.” “Asking questions” was a frequent communication mode used by all, with providers asking questions at twice the rate of caregivers.

Surgical decision-making:

Applying a qualitative thematic analysis of provider-caregiver discussions, clinic-based discussions included, in descending order of frequency: “option of not doing surgery,” “surgery is not urgent,” “surgery is cosmetic/elective,” “surgery is controversial” or “future interest of child,” and “consultation with outside sources” (eg, peer support or advocacy organization). Parents and providers used ambivalence and uncertainty about interventions and potential outcomes to selectively attend to details and discount warnings. We did not observe substantive differences in communications before vs after DST introduction.

DST use:

Few caregivers used the DST as designed during pilot testing: 12% of caregivers reviewed all content and completed decision-making exercises; 40% used portions; and 48% did not use it past initial logins at clinic. Despite direct involvement in constructing the DST and participating in a study of its utility, providers infrequently referred to the DST during clinic visits.

Conclusions and Limitations:

Despite calls for an interactive web-based DST, few families used the tool as intended, and few clinicians mentioned its use during clinic encounters. Specific training in integrating the DST in the context of usual care is needed to increase its use, which, in turn, creates the opportunity to assess its value in structuring communications that reflect the shared decision-making process.

Background

Introduction to the Condition and Challenge

Differences (or disorders) of sex development (DSD) is an umbrella term that encompasses congenital conditions in which chromosomal, gonadal, or anatomic sex development is atypical.1 Clinical management decisions (some irreversible) on behalf of newborns and young children with DSD are often made at times when parents feel intense emotional distress and struggle with learning about their child's complex medical condition. Anxiety-driven decisions often reflect perceptions of a limited range of options with inadequately weighed risk and benefit potentials.2-7 The “right” decisions about best courses of action are not always obvious. While major advances in diagnostic assessment (eg, genetic testing) have occurred, results do not always reveal the cause of the DSD. Further, even when the tests uncover the responsible genetic variation, that knowledge does not commonly lead to a single “correct” treatment plan. That is, a diagnosis (even a molecular genetic diagnosis) may suggest many different clinical management options, each of which can be associated with a positive outcome.8,9 With the assistance of the child's medical providers and others, parents need to make decisions based, in part, on their knowledge, preferences, values, and culture.10,11 However, former patients and health care advocates complain that parents are provided inadequate information with which to make decisions for their child. Research corroborates the claim that parents have often not heard of DSD conditions before their child's birth and typically do not perceive that treatment choices exist; instead, many assumed the diagnosis implied a single, specific treatment path.12

Although conditions encompassed by the term DSD are diverse in terms of pathophysiology, they share many of the same factors that have been shown to exert profound effects on family adjustment, parenting, and emerging self-concept of the affected person. Decisions faced by families and health care providers who care for children affected by DSD may include gender of rearing; pros and cons of next-generation genetic sequencing; surgery (genital and/or gonadal) and its timing; if, how, and when to talk to others (eg, extended family, close friends, the child himself or herself, and siblings) about the DSD; contacting peer support and/or advocacy organizations; and others.13,14 Although justified as therapeutic, surgery in DSD is largely elective and, with limited exceptions, irreversible decisions could be postponed to an age when the minor is competent to be involved in discussions and provide assent without risking threats to physical health.15-17 An issue here is whether performing genital surgery late in life, or not at all, is associated with poorer psychosocial and psychosexual outcomes than performing these procedures early in life. Increasing emphasis and demands for patient-centeredness in health care delivery have been accompanied by controversy in DSD care, especially regarding elective genital or gonadal surgery in infancy or early childhood. Legal and ethical questions have been raised—including a recent lawsuit broadly covered in the media18-21 and litigated by the respected Southern Poverty Law Center22—regarding the legality of early surgical interventions in DSD. Some patient advocates, equating surgical intervention to “torture,” have worked with the UN High Commissioner for Human Rights and the UN Special Rapporteur on Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment to call for the “prohibition of surgery and treatment on the sex characteristics of minors without informed consent.”23 There are also newly emerging concerns about the safety of anesthetics administered to infants and children younger than 4 years on neurocognitive development.24 To the extent that complexities inherent in DSD-related clinical management decisions are not routinely and systematically presented to parents, the likelihood of decisional regret is increased.25

Parents of young children with DSD are responsible for making decisions on behalf of their children. However, they often do not know that there are decisions to be made nor that they play important roles in a shared decision-making (SDM) process. For example, a qualitative study of parental experiences revealed that many parents viewed genital surgery to “normalize” their child's sexual anatomy as an obvious and necessary clinical management component; that is, for them, there was no point at which any decisions needed to be made regarding genital surgery.12 The objective of SDM is to help patients (or surrogates; ie, their parents) make informed, preference-based clinical management choices among several relevant options.25 SDM does not imply that doctors and patients must have equal responsibility for the final decision.26 Rather, SDM combines health care providers' expert knowledge and patients' (or surrogates') rights to make health care decisions with full information; it requires the involvement of both health care providers and the patients/parents, with bidirectional information exchange, mutual deliberation on treatment options, and agreement on treatment plans.27,28

As patient and provider stakeholders have voiced concerns that families may not have all the information necessary to make informed decisions, the overall objective of this project was to develop a decision support tool (DST) for caregivers to navigate complex and controversial clinical management decisions on behalf of their young children. More specifically, this study set out to determine (1) the content and delivery format for a DST desired by stakeholders (eg, parents of affected children, affected adults, patient advocates, pediatric endocrinologists, geneticists, gynecologists, psychologists, urologists/surgeons, ethicists, counselors, and others); (2) the communication elements (eg, content, structure, and tools used by families and health care providers) characterizing standard-of-care appointments at DSD clinic—both before and after introducing the DST into regular care; and (3) what is needed to fully integrate the DST into regular clinical management from the perspective of all stakeholders. That is, what characteristics of the DST and the health care system affect its likelihood for implementation?

Significance and Potential Impacts of the Research Envisioned at the Time of the Award

Prompted in part by a “heightened awareness of ethical issues and patient advocacy concerns,” a consensus conference on the management of DSD was convened in 2005.1 More specifically, a significant factor contributing to convening the conference was disaffection among former patients, patient advocates, and some members of the medical community regarding the model of care delivered to affected persons and their families.1,29,30 Perhaps nowhere was this dissatisfaction more profoundly experienced than in irreversible surgical decisions made early in the affected child's life.31-34 The gaps in knowledge of long-term outcomes and in research elucidating factors that bias developmental trajectories toward better or worse health-related quality of life outcomes have continued and serve as a call for improved communication among parents and health care providers involved in diagnosing and managing DSD. However, this same set of deficiencies is likely responsible for quick clinical decisions being taken based on beliefs or unfounded assumptions about what is “medically necessary” to achieve positive long-term outcomes for the child.17 The science of medical decision-making has matured rapidly, but the products of this research directed at parent proxy decision makers for young children and their health care providers has been limited, and none have yet been developed for DSD.35 Decisions that parents need to make on behalf of their children start at the point of ascertaining that somatic sex development has followed an atypical pathway. Increasingly, this process begins during prenatal life when, for example, a discordance between karyotype (ascertained by amniocentesis) and genital appearance (imaged by ultrasound) can initiate the DSD counseling process even at this early stage.36 The availability of a DST that informs the parents about DSD in general and the process of diagnosis and treatment options, and identifies their family and cultural beliefs, values, and preferences that are important to them and affect their decision-making, will prepare parents to become fully involved in management decisions on behalf of their young children.

Participation of Patients and Other Stakeholders in the Design and Conduct of Research and Dissemination of Findings

Phase 1—DST Development

Forty-nine stakeholders assisted the research team with developing the DST's content and format. These included 15 parents of young children affected by DSD, 11 leaders of patient advocacy organizations, and 23 health care providers.

We targeted parents for participation following 2 different procedures involving medical centers (n = 8) and resource and advocacy organizations (n = 7).

Medical Center Recruitment

We identified parents via consecutively referred cases to 1 of the 3 participating medical centers. We reviewed medical charts of young children receiving clinical services for the assessment and/or management of DSD to identify parents of affected children who met eligibility criteria. A member of the research team offered participation.

Resource and Advocacy Organization Recruitment

We facilitated recruitment of parent members of support group and advocacy organizations through a partnership with Accord Alliance, a convener of all stakeholders interested in improving the quality of care for people with DSD and their families.37 Accord Alliance distributed the study invitation letter to support group and advocacy organization representatives as well as parents with a child affected by DSD who were affiliated with the same organizations. Group/organization members were also invited to nominate other members (ie, snowball technique). Accord Alliance coordinated recruitment of resource and advocacy organizations' leaders.

We targeted specialist health care providers (n = 16) based on their experience in managing DSD, as evidenced by peer reviewed publications on DSD, discipline-specific DSD committee work, and leadership responsibilities in their respective professional societies. Additionally, this project took advantage of an existing US clinical and research infrastructure for recruiting health care providers: the DSD-Translational Research Network (DSD-TRN).38 The DSD-TRN comprises interdisciplinary health care teams—growing from 4 to 10 pediatric academic medical centers during the period of the PCORI award. Health care specialists included geneticists, genetic counselors, pediatric endocrinologists, urologists, gynecologists, behavioral health providers, bioethics experts, and primary care providers (PCPs). We sent clinicians representing the above-mentioned specialties from the DSD-TRN sites an email explaining the project and inviting them to participate in reviewing the DST during its development, to provide feedback about content and form. Finally, we interviewed the primary care physicians (n = 7) of chart-selected DSD patients to learn about their views on their role in DSD management and their opinions on the DST.

Initially, we interviewed stakeholders to learn about their background and to provide guidance on the content and format of the DST. We created a content document that summarized findings from the literature and initial interviews and documented the guiding principles, content, and format of the DST. We shared this document with stakeholders and collected feedback. From there, we drafted a DST, shared it with stakeholders, and edited it based on feedback in an iterative fashion until we completed a provisional DST. Interactions occurred in person, by telephone, via email (ie, sending out copies of proposed content and receiving written feedback), and over the internet (eg, when access to the various iterations of the web-based DST was granted so stakeholders could examine it and provide feedback). Additionally, for members of the research team, health care providers, and key leaders of the patient advocacy and organizations, we created a password-protected website to track and display project progress, house documents associated with the DST (eg, proposed content), and allow comments to be posted.

In this fashion, stakeholder involvement was essential to shaping the DST's content and format. Ideally, all stakeholders would agree on all aspects of the DST; however, differences of opinion emerged regarding the depth and complexity of information (eg, geneticists preferred lengthy, detailed information; advocates preferred only the “essentials”) and middle grounds needed to be struck using all available information from these stakeholders in conjunction with extant research literature and findings from early audio-recorded clinic appointments.

Phase 2—Examining Family-Provider Communication During Clinic Visits

Sixty-three parents/caregivers of 31 young children who were seen at 1 of 3 DSD clinics agreed to allow the research team to audio-record their interactions with providers during their clinic appointments. Providers (n = 60) included specialists in endocrinology, genetic counseling, genetics, gynecology, nursing, psychology, social work, and urology. We identified index cases via a review of medical charts for those with upcoming appointments at 3 medical centers. All potentially eligible participants (ie, the adult caregivers of children aged 0 to 5.9 years being seen for the assessment and/or management of DSD in which at least 1 major clinical management decision needed to be made on behalf of the child existed) were informed about the study and offered participation. Caregivers included biological, adoptive, and stepparents; grandparents; and other family members who were regularly involved with the child's care. Thirty-six caregivers participated before the creation of the DST; 25 different caregivers participated after its creation and were provided access to it. Use of the DST among parents who were provided access varied across and within families.

Stakeholder engagement before releasing the provisional DST for use in clinic was essential to better understand what content elements needed to be included and the format (including word-choice and health-literacy concerns) in which information needed to be presented. After sharing the DST with families and providers in clinic, audio-recorded data collected in clinic provided clues regarding uptake and use (or nonuse) in the context of regular care. The research team was able, at times, to make on-the-fly adjustments to accommodate family needs (eg, printing out sections of the DST for families to review offline; creating hard-copy flowchart illustrations for providers to use with families that indicated which portions of the DST were most highly recommended for families to use based on the specific child's diagnosis, family characteristics, and overall clinical management plan).

Phase 3—Identifying Opportunities and Barriers to DST Implementation

Forty stakeholders provided feedback on the provisional DST that we developed for use in clinic. This included 11 parents of affected children, 4 resource and advocacy organization leaders, and 25 health care providers (2 of whom were the primary care physicians of chart-selected patients whose parents participated in the audio-recorded clinic visits; 23 of whom were recruited independently—recruitment methods noted above). All participants were provided access to the online DST and provided feedback via web, email, and/or telephone interview.

Methods

This project proceeded in 3 phases: DST development, examining family-provider communication during clinic visits, and identifying opportunities and barriers to DST implementation (see Figure 1).

Figure 1. Project Phases.

Figure 1

Project Phases.

Phase 1—DST Development

Purpose of This Phase

The goal of this phase was to develop a DST that reflects the full range of parental needs in both the types of information and the format in which it is delivered, in a manner sensitive to their educational, emotional, cultural, and other needs. The DST also had to be acceptable to health care providers who work with these families.

Methods for This Phase

The research team used guidance and feedback from multiple stakeholders on the development of successive drafts of (1) the content document that delineated the principles and goals of the DST; and (2) preliminary versions of the DST's content, format, and suggested integration within routine clinical care (Table 1).

Table 1. DST Development.

Table 1

DST Development.

Content document

A content document specifies the project's principles and goals. The research team generated a rough draft based on the research literature and their own experiences, then shared it with stakeholders by email. Stakeholders—who were recruited via snowball sampling, beginning with recommendations of research team members—provided feedback via track changes editing and comment bubbles in Microsoft Word, comments recorded by the research coordinator during phone calls, and written email replies. The research coordinator collated feedback, shared the collated lists with the research team and stakeholders, then revised and shared the document with the rest of the research team and stakeholders in an iterative fashion until a final version was produced that all agreed on (7 iterations). We structured DST development according to the recommendations of the Consensus Statement on Management of Intersex Disorders1 and the Ottawa Decision Support Framework to Address Decisional Conflict (Ottawa),39,40 which (1) follows International Patient Decision Aid Standards41 and (2) is particularly relevant to preference-sensitive decisions.42,43 The Ottawa framework includes 3 key elements: (1) assessment of determinants of decisions (both patients' and providers' decisions); (2) provision of decision support interventions to prepare the patient/family and provider to make and implement a decision; and (3) evaluation of the success of the interventions at improving the quality and outcomes of the decision process. Additional detail is also provided to define determinants of decisions, such as sociodemographic and clinical characteristics, patients'/families' and providers' perceptions of the decision and what important others think about the decision, and resources (both personal and external) available to make the decision.

Preliminary versions of DST

Development proceeded in 4 stages: (1) outline and PowerPoint slides; (2) preliminary DST, version 1; (3) preliminary DST, version 2; and (4) provisional DST (see Table 1). Following methods and standards used in developing the content document, the web-based DST was similarly created via an iterative process of drafting materials, receiving feedback from a wide range of stakeholders (parents of affected children, specialty providers, and patient advocates), editing materials, and seeking additional feedback until an online DST was established. We coded all feedback and sorted it into themes. Informal “member checking”44 of these themes occurred via email or phone exchanges, with more formal checks occurring as edits, based on the feedback, were subsequently shared and revised further as needed.

To address literacy, we subjected the text of each section under development to a readability analysis, with an emphasis placed on targeting a sixth-grade reading level on the Flesch-Kincaid scale, which is implemented in Microsoft Word. Given the complex nature of the material, we rarely, if ever, achieved this goal. For example, the single word “endocrinologist” in a sentence by itself merits a 12th-grade reading level rating. We consulted additional guidance on a regular basis from the Michigan Medicine patient education and health literacy program lead,45 and resources including the Plain Language Alternatives for Patient Information and Consent Materials46 and Plain Language Thesaurus for Health Communications,47 to guide selection of alternative language. The research coordinator introduced simplified language via edits, then circulated edited copies for review (and continued editing) until stakeholders approved a finalized version.

Population/Participants

Recommendations stemming from the 2005 Consensus meeting appeared the following year in a published consensus statement; these included the following:

optimal clinical management of individuals with DSD should comprise … an experienced multidisciplinary team … open communication with patients … [in which] participation in decision-making is encouraged.1

As such, the research team identified and sought participation from the following stakeholders: parents of affected children, patient resource and advocacy leaders and members, specialty and PCPs, bioethicists, and others. We recruited parents of affected children either from national resource and advocacy organizations (RAOs) or through hospitals via chart selection. Of note, representatives of RAOs are a self-selected group: Research in the area of peer support suggests that only a small minority of patients/caregivers actively participate in peer support organizations14 and that they may not be representative of the population of people meeting criteria for the medical condition.48 Nevertheless, in the area of DSD care, RAOs represent a powerful voice that characteristically adopts positions that are in opposition to historical standards of care for DSD; thus, we purposefully sampled parent members and leaders to ensure a broad range of perspectives were heard. Also, as noted, to ensure that the voices of a minority were not overweighed, we adopted an additional strategy to tap caregiver perspectives: identification through medical chart selection. This approach had the added value of making it possible to stratify recruitment to ensure representativeness across the full range of DSD conditions.

Recruitment procedures for parents identified through review of affected children's medical records

The IRB granted a waiver of informed consent for case ascertainment. To ensure a representative sample, the principal investigator (PI) and/or designee conducted a medical chart review. Once we established that a family met the criteria (see Table 2), we invited parents to participate either by mail or in person if they happened to be attending the DSD clinic during the recruitment period. We emphasized that decisions regarding participation would not affect the medical care provided to their child. In person, the PI/research staff designee explained the project and solicited written informed consent. By mail, we sent a recruitment package that included an invitation letter, informed consent documents, a sheet to collect demographic information and availability for interview scheduling (to be returned if the family agreed to participate), and a postage-paid “do not contact” postcard (to be returned if the family declined to participate).

Table 2. Study Participants: Inclusion and Exclusion Criteria; Recruitment Procedure.

Table 2

Study Participants: Inclusion and Exclusion Criteria; Recruitment Procedure.

The PI contacted leaders of RAOs to assist with RAO member parent/caregiver recruitment. Families were then contacted by the leadership of their respective RAO. This contact occurred by the means each organization regularly used to contact members. Members were given information about the project and asked to contact the PI/designated research staff via the method of their choosing (eg, phone; email; through the RAO, who would then pass information along to the research team). The PI/designee then described the project further with the caregiver to ensure that elements of informed consent were covered and eligibility requirements were met (ie, parent of a child who has DSD, comfortable with written and spoken English, access to phone/means of providing input), and gathered available times to schedule a group or individual interview.

Specialist health care providers and RAO leadership members were not considered to be participants in human subjects research because their participation was limited to feedback on a developing product; ie, the DST. Thus, the IRB waived formal consent procedures. A description of inclusion/exclusion criteria and first point of contact for each stakeholder group is found in Table 2.

Setting

Stakeholders reviewed materials at their leisure and in a setting of their choosing. They were, similarly, free to provide written or verbal feedback from the setting of their choosing.

Data Collection and Sources

We collected feedback in person, by telephone, via email (eg, sending out copies of proposed content and receiving written feedback), and over the internet (eg, when access to the various iterations of the web-based DST was granted). We also utilized parent focus groups: 15 parents were recruited through clinic (n = 8) and RAOs (n = 7), 12 of whom participated in a focus group. Focus group participants were provided access to the web-based DST. Parents participated in 1 of 4 focus groups about their experiences in using the preliminary DST. Semistructured interview questions focused on opinions about the content (eg, informational content, values identification exercises, factors in decision-making), delivery (eg, presentation, length, desire to work with health care providers), and appearance and usability (eg, format, navigation). We audio-recorded, transcribed, and analyzed interviews to identify recurring topics (eg, general user acceptability, opinions on the amount and type of information presented, values identification). Several participants also followed up via email. Additionally, for members of the research team, health care providers, and key leaders of RAOs, we created a password-protected DST-project-specific intranet site to track and display comments regarding proposed content. We transcribed verbal feedback. We stored written feedback on the website for others to view.

Outcomes

The main outcomes of this phase of research were (1) the content document and (2) the DST.

Analytical Approach

We audio-recorded and transcribed qualitative feedback collected in the form of group and/or individual interviews and coded them for responses to questions pertaining to topic coverage/content, presentation, and usability. Additionally, interviews were coded for the presence of recurring comments/themes unanticipated by researchers. We similarly coded written feedback for pertinent themes. The research coordinator collated and presented these data to the research team via email; site PIs and patient resource representatives then discussed these data in the context of weekly leadership group calls, to form the final set of themes used in the early portion of the project and to guide DST development within the Ottawa39,40 framework. We addressed differences in opinion and contrasting recommendations (eg, some health care providers recommended detailed explanations of genetic aspects of DSD vs patient advocate recommendations to include only the essentials) by including the contrasting views in the collated documents, presenting to the research team, and relying on consensus through discussion. The PI ultimately resolved any unresolved points by referencing the Consensus Statement,1 the Ottawa framework,39-40 and health literacy guidance.45-47

Phase 2—Examining Family-Provider Communication During Clinic Visits

Purpose of This Phase

We originally conceptualized this phase as 2 stages: (1) assessment of usual care and (2) piloting of the DST. Other than the availability of the DST for use (available only during stage 2 piloting), methods were identical. As such, the description of the 2 stages is collapsed into examining family-provider communication during clinic visits. The purpose of this phase was 2-fold. First, a better understanding was needed of the content elements to be included and the format (including word-choice and health-literacy factors) in which information needed to be presented. Second, after sharing the DST with families and providers in clinic, audio-recorded data collected in clinic were needed to provide clues regarding uptake and use (or nonuse) of the DST in the context of usual care.

Methods for This Phase

We audio-recorded regular appointments in DSD with the goal of capturing in-clinic caregiver-provider discussions. Additionally, we asked caregivers to complete several short validated paper-and-pencil questionnaires to measure factors related to decision-making: Decisional Conflict,49 Decision-Making Preference,50 Patient Health Screeners, the Generalized Anxiety Disorder-7 (GAD-7)/Patient Health Questionnaire-9 (PHQ-9) that screened for anxiety and depressive symptoms,51 Relational Communication,52 Source Credibility,53 and a modified Trust in Providers.54,55 We also asked caregivers to participate in a semistructured interview about their thoughts on developing a DST, or feedback on the DST, depending on whether they participated before or after its creation.

Population/Participants

Recruitment eligibility depended on the index child's age (newborn to 5.9 years), if they received care at 1 of the 3 DSD clinics for either the assessment or ongoing clinical management of DSD, and if they had at least 1 major clinical management decision to consider. Clinical management decisions comprised those that naturally arose during the course of clinical care depending on the nature of the child's condition; ie, they were not determined by the research protocol. As such, decisions may include gender of rearing; single gene vs whole exome sequencing; genital and/or gonadal surgery and its timing; if, how, and when to talk to others about the child's DSD; and others that arose for families. Adult caregivers included biological, adoptive, and stepparents; grandparents; and other family members who were regularly involved with the child's care. They were informed about the study and offered participation, and, if interested, they provided written informed consent. We assured all potential participants that their decision would not affect the care provided to them or their children. The site PI contacted specialist health care providers who were members of the DSD Clinic team at each site in advance of case ascertainment to gauge their interest in participating. Providers were given an informed consent document to review with a research staff member. Because the research protocol followed only regular care (ie, providers were not selected to see patients/families based on interest in participating), health care providers were allowed to choose to participate in the audio recording on a case-by-case basis: Each provider was given a recorder; the audio recording acted as their consent for a case. Some health care providers at each site had also provided feedback about the developing DST.

Setting

Audio recordings of clinic appointments occurred in the context of regular clinic visits. Questionnaires were administered close in time to recruitment at the site of recruitment (ie, typically at clinic) or in the context of telephone interviews.

For those caregivers who participated after the DST was created, the research coordinator provided website access at clinic in conjunction with the clinical coordinator; clinical coordinators are members of the patient's health care provider team. The research coordinator determined sections of the DST of particular relevance on a case-by-case basis via discussion among the patient's providers (ie, the clinical coordinator and subspecialty providers who regularly provided patient care) and recorded them. Parents received a list of DST sections of particular relevance to their child's clinical care, listing areas on which they should focus. Because the web-based DST had a mechanism that tracked when a study participant completed various activities within the DST, the research coordinator monitored progress and informed clinicians of the caregiver's usage so that the clinician could incorporate that knowledge into their clinical management.

Data Collection and Sources

Clinic appointments were audio-recorded beginning at the time of recruitment and continued until a clinical management decision was made. Decisions varied in terms of what clinical management decisions were present (eg, decisions about genetic testing or decisions about surgical options) and what choices were made (eg, to decide for or against or to defer decisions until the child is older) for each case. After clinic recordings were complete, caregivers were asked to talk with a member of the research team about their experiences and share their opinions on the pros and cons of a DST (if they participated before DST use in clinic; ie, phase 2, stage 1; n = 36 caregivers; patient age range, 0.01-5.54 years; mean, 1.0 years) and experiences with the DST (if they participated once it was in place; ie, phase 2, stage 2; n = 25 caregivers; patient age range, 0.01-5.17 years; mean, 2.3 years). We transcribed audio recordings for subsequent analysis. We administered questionnaires at 2 time points: close in time to the initial set of recordings (ie, before clinical management decisions were made) and during the postdecision interview with research staff (ie, after decisions were made). Initial questionnaires used paper and pencil; follow-up questionnaires were administered as fully structured interview questions.

Outcomes

Primary outcomes included identifying the following: DST content and delivery mechanisms not previously identified through initial interviews and feedback on successive drafts; topics and decisions discussed by caregivers and providers; instances in which decision-making opportunities were present but not discussed; and communication style (ie, how the information was discussed—eg, were closed or open-ended questions posed) used by caregivers and providers. We also made comparisons between stakeholders (eg, parent vs specialist and between specialty providers).

Analytical Approach

Given the vast amount of audio-recorded and transcribed materials, we analyzed qualitative data (ie, audio recordings of clinical encounters) using the Siminoff Communication Content and Affect Program (SCCAP).56 SCCAP facilitates, organizes, and codes health communication data and is designed so that raters can code from audio recordings. Specifically, the program captures (1) task-driven information exchange between multiple interaction partners, (2) the affective and relational communication activities of all communicators, and (3) the persuasion and compliance tactics employed in health care settings that contribute to decision-making. In this study, communicative content and the use of relational techniques are coded as binary variables. Counts, frequencies, and ratios can be employed in traditional statistical analyses. For example, we compared how frequently and how much time was spent on certain content issues, the ratio of family decision maker to health care provider speech, and the number and types (open ended or close ended) of questions asked by family decision makers. Additionally, we coded many of the elements derived from the parent self-report questionnaires using SCCAP methods. This multimethod approach resulted in filling gaps from questionnaire data: The number of questionnaires completed did not result in sufficient power to justify quantitative analyses.

Additionally, we coded clinic-based transcript data to distinguish the arguments in favor of and against surgery and to map the course of decision-making.57 This process of qualitative data analysis began with reading the transcripts iteratively in light of the social science literature on patient-doctor interaction, medical decision-making, and DSD. Two coders independently reviewed transcripts from all cases to identify all decisions about surgery and, once complete, grouped interactions by the parents' initial inclination toward surgery. The coders reconciled all discrepant codes through discussion. The analysis sought out negative cases58,59 to strengthen the emerging conceptualization. This involved coders rereading transcripts and searching for data that either did not support or appeared to contradict patterns or explanations that emerged from data analysis.

Phase 3—Identifying Opportunities and Barriers to DST Implementation

Purpose of This Phase

This stage of the project occurred after we had created and shared with stakeholders the provisional DST for use in clinic; it focused on elements important to dissemination efforts—with particular emphasis on how the DST could continue to evolve and be used outside the confines of a research study (Figure 1).

Methods for This Phase

We shared the DST with stakeholders via one of several mechanisms (see the “Population/Participants” and “Data Collection and Sources” sections below). We obtained feedback on the DST via interviews, meeting notes, email, and other written communications (“Data Collection and Sources” section). We recorded and coded feedback (“Analytical Approach” section below).

Population/Participants

We recruited RAO leaders and specialist health care providers as in previous phases. Their feedback was joined by parents of affected children who participated in phase 2, examining family-provider communication during clinic visits (previously described), and PCPs. PCP recruitment hinged on family involvement in phase 2. We contacted PCPs only after participating caregivers provided written consent for the research team to contact their child's PCP. We sent an introductory letter in the mail with a copy of the informed consent that the family signed so the PCP was aware of the details of their patient's participation. Because PCPs were geographically dispersed, we obtained informed consent by phone and had no face-to-face contact; these providers' participation was limited to commenting on their experiences as a PCP to patients who have specialty care physicians; ie, similar to conversations they typically have on listservs and at professional meetings.

Setting

Stakeholders reviewed materials at their leisure and in a setting of their choosing. Participants were, similarly, free to provide written/email or verbal/interview feedback from the location of their choosing.

Data Collection and Sources

Data consisted of stakeholders' written feedback and audio-recorded interviews that we subsequently transcribed. Aside from PCP involvement, in which the goal was to recruit all PCPs with whom families allowed contact, recruitment and data collection from all other stakeholders continued until saturation (ie, the point at which no new data/themes emerge despite the addition of more participants) was reached. PCPs (n = 2) for young patients seen in clinic whose parents were involved in the project were provided the link to web-based DST and participated in semistructured interviews that targeted experiences working with patients and families with DSD, DST presentation/format, actual and anticipated DST use with affected families, and advice for implementing a DST—in particular, the role of primary care.

Patient advocates and specialist professionals were also provided a link to the DST and a link to a short questionnaire about the DST (n = 22 health care providers, n = 4 advocates) that focused on the ease of logging in, the design and content of each major section, and the extent to which they believed the support tool would support parents with a variety of (specified) decision-making tasks. A subset of these individuals (n = 16 health care providers, n = 4 advocates) also participated in a semistructured interview that targeted their view of the degree to which core decisions/key dimensions were included in the DST, terminology and balancing factually correct information without oversimplifying or making it too difficult to understand, who should use/administer the DST, anticipated barriers to implementation, and recommendations on how to measure outcomes of DST use in the future. Finally, DST use, as seen via online usage statistics, was recorded for caregivers who were provided access to the DST during phase 2, stage 2, examining family-provider communication during clinic visits.

Outcomes

The primary outcome was a qualitative analysis of feedback regarding DST use and delineation of future directions regarding its implementation and dissemination.

Analytical Approach

As in earlier phases, we transcribed and coded qualitative feedback (audio-recorded interviews) for pertinent themes regarding recommendations offered by participants. We similarly coded written feedback. As in phase 1, the research coordinator collated these data and presented the results to the research team.

Results

Participants

Table 3 summarizes the numbers of participants by stakeholder group.

Table 3. Description of the Study Sample Size Throughout the Project.

Table 3

Description of the Study Sample Size Throughout the Project.

Phase 1—DST Development

Fifteen parents of affected children, 11 patient advocates, and 23 health care providers participated during the development of the web-based DST for use in clinic. Parents of affected children participated in 1 of 4 group interviews with approximately equal numbers (8:7) recruited from each source (ie, clinic or patient/family resource and advocacy organization). Patient advocacy leaders came from several different patient advocacy organizations. Health care providers included PCPs and specialty health care providers representing endocrinology, genetics, genetic counseling, psychology, and urology.

Phase 2—Examining Family-Provider Communication During Clinic Visits

Of the 63 parents/caregivers whose interactions with providers during their clinic appointments were audio-recorded, 36 participated before the creation of the DST and 27 participated after its introduction. One patient was determined not to have DSD during the course of the assessment; thus, results from 25 of 27 (or 61 of 63) are included (see Table 3). Most identified as female (n = 35 [57.4%]), White (n = 42 [68.8%]), and of non-Hispanic descent (n = 49 [80.3%]; see Table 4). Health care providers included representation from endocrinology, genetics, genetic counseling, gynecology, nursing, psychology, social work, and urology/surgery.

Table 4. Participant Characteristics: Race, Ethnicity, and Sex of Parents of Affected Children Participating at Clinic Visits.

Table 4

Participant Characteristics: Race, Ethnicity, and Sex of Parents of Affected Children Participating at Clinic Visits.

Phase 3—Identifying Opportunities and Barriers to DST Implementation

In addition to caregivers who were given access to the DST during the examining family-provider communication during clinic visits phase—10 of whom participated in interviews to collect feedback regarding DST use—4 RAO leaders and 25 providers provided feedback on the DST. Providers included PCPs and specialist health care providers representing endocrinology, genetics, obstetrics/gynecology, psychology, social work, and urology. More specifically, this phase took advantage of an existing clinical and research infrastructure in the United States, the DSD-Translational Research Network, that consisted of DSD teams across 11 hospital sites in the United States providing care to families affected by DSD.38 Stakeholder groups contacted were geneticists, genetic counselors, pediatric endocrinologists, urologists and gynecologists, behavioral health and bioethics experts, and PCPs/medical homes. Of the 128 people screened, 54 medical providers were invited to provide feedback on the web-based decision aid (ie, they did not participate in other development phases of this project). Participants were given a personal login code to access an online survey about the design and content of the tool. Only after the survey was completed were participants (n = 22) contacted for a follow-up interview. Thirty medical providers initially agreed to participate; however, only 16 providers actually participated, despite numerous follow-up contact attempts. In addition, 15 representatives of support group and advocacy organizations were screened, of whom 7 did not provide feedback in previous feedback rounds and contacted through the Advisory Advocacy Network—a unit within the DSD-TRN. Five representatives initially agreed to participate, of whom 4 actually participated.

Outcomes

Phase 1—DST Development

Stakeholders identified the following as essential ingredients, which formed the basis of the content document: (1) providing information about treatment options and associated risks and benefits; (2) presenting the likelihood of varying diagnostic tests (biochemical and genetic, imaging studies, exploration under anesthesia) and their timing; (3) interpreting “incidental” genetic findings and carrier status; (4) interpreting test results and their relationships to gender assignment decisions; (5) eliciting caregiver values and preferences; (6) using personal stories/testimonials; (7) coaching caregivers in effective communication with the health care team; (8) disclosing conflicts of interest; (9) providing balanced information about treatment options; (10) addressing health literacy; and (11) delivering decision support on the internet. The content document, in turn, delineated the principles and goals of the DST.

Stakeholder feedback regarding successive iterations of the provisional DST varied over time, in response to edits to (and leading to further editing of) DST content, format, and delivery platform (see Table 1). Qualitative analysis of the feedback led to identification of several themes that primarily dealt with issues commonly seen in human factors engineering, the scientific field that focuses on creating systems based on user-centered designs.60,61 Specifically, themes centered on accessibility, language, structure, and navigation. The ultimate goal of this phase of research, however, was the creation of a web-based DST.

We created the provisional DST for use in clinic, DSD Support Tool: A Guide for Parents, found at DSDSupport.org (no longer functional). The site itself was a WordPress site that presented content and IFramed versions of Qualtrics forms. An IFrame (Inline Frame) is an HTML document embedded inside another HTML document on a website; IFramed documents behave like inline images, however, they can be configured with their own scrollbars independent of the surrounding page's scrollbar. The Qualtrics forms enabled interactive data collection. Major domains/modules included the following:

  • Introduction (Welcome, an interactive questionnaire that assesses knowledge about DSD before DST use; What Is DSD; and Sex Development Basics)
  • Evaluation/Support (an interactive survey on Sharing With Others, Support for Families, Evaluation of a Baby With a DSD, Genital/Physical Exams and Medical Photography)
  • Testing (What Tests Are Usually Done First?, Extra Tests, a survey on Genetic Testing)
  • Your Child's DSD (educational materials and questionnaires intended to help families recognize occasions when decision-making opportunities exist; identify family beliefs, values, and preferences; and record current status on decision-making in these areas)
  • Assigning a Gender of Rearing (educational materials and questionnaires intended to help families recognize occasions when decision-making opportunities exist; identify family beliefs, values, and preferences; and record current status on decision-making in these areas)
  • Surgery (educational materials and questionnaires intended to help families recognize occasions when decision-making opportunities exist; identify family beliefs, values, and preferences; and record current status on decision making in these areas)
  • More Information (well-being assessments, parents' stories, an interactive questionnaire that tests knowledge about DSD after reaching this point in the DST, a list of Questions to Ask Your Child's Health Care Providers, Additional Resources, and a Glossary).

The web-based DST opens with an introduction module seen by all users. Overall, the introduction conveys basic information about DSD (definition, prevalence, and risk factors); that gender is not directly determined by sex chromosomes, gonadal histology, endocrine function, or reproductive anatomy; and that early decisions, some of which are irreversible, have lifelong implications.25 In addition, the emotional and social impact of these conditions, support opportunities, and diagnostic evaluation process are elucidated. Simplified information is seen by all users, with the option to take a deeper dive into a given topic via hyperlinks. Next, parents are given more specific information about their child's DSD diagnosis or descriptive features (if no diagnosis is yet available) and (medical) treatment, and are assisted to identify the immediate decisions they specifically face regarding gender of rearing, gonadal management, and genital management. Printer-friendly information graphics are used to complement the text as graphic visual representations of the information, data, or knowledge. Families are next asked to consider other issues important in the decision and to clarify their family values using a weigh scale to rate the perceived importance of certain values and decisions. They are also asked to identify their preference for participation in decision-making and indicate their predisposition or “leaning” toward certain decisions. A summary of their value and decision preferences can be printed out or electronically transmitted to the DSD team for further discussion. Parents are also asked to list their questions (via a preidentified question prompt list) to be discussed in follow-up visits with health care providers. Further clinical encounters include reviewing possible benefits and risks with providers to verify and fill in gaps; discussing personal values by showing the practitioner the weigh scale; making a decision considering benefits, risks, and personal values; and planning the next steps. At present, these materials can be found in workbook form, available from the PI (see Figures 2-4 for samples of the DST workbook pages).

Figure 2. DST Workbook: Opening Pages.

Figure 2

DST Workbook: Opening Pages.

Figure 3. DST Workbook: Information Pages—Selected Examples.

Figure 3

DST Workbook: Information Pages—Selected Examples.

Figure 4. DST Workbook: Worksheets—Selected Examples.

Figure 4

DST Workbook: Worksheets—Selected Examples.

Phase 2—Examining Family-Provider Communication During Clinic Visits

We audio-recorded interactions between caregivers and providers at 3 medical centers to capture caregiver-provider discussions of clinical management decisions. Recordings took place before and after a clinical management decision had been made. Decisions differed by index case; eg, whether to purse genetic testing, whether to pursue a surgical intervention. Two centers followed a multiprovider communication model in which family members met with several providers simultaneously for 1 to 2 meetings lasting 9 to 123 minutes. The third center had serial parent-specialist discussions, each lasting 2 to 96 minutes. DSD team providers included pediatric endocrinologists, geneticists/genetic counselors, psychologists, and urologists/gynecologists. Most appointments took place within the framework of the DSD clinic. Slightly more than half (n = 36 [57.1%]) of family caregivers participated before the creation of the DST and 25 participated after its introduction.

Caregiver-provider conversations

Overall, health care providers spoke more than caregivers. Based on the SCCAP analysis, of 54 053 utterances, 30 751 (56.9%) were health care provider utterances and 23 302 (43.1%) family utterances, yielding a ratio of 1:0.76. Comparing by specialty, geneticists and genetic counselors accounted for 24.1% (n = 13 017) of the total utterances, endocrinologists 11.4% (n = 6150), urologists/gynecologists 10.8% (n = 5828), psychologists 9.7% (n = 5267), and other health care providers 0.9% (n = 489) (P < .001). The most frequent topic was “medical information,” accounting for 24.3% (n = 13 122) of total utterances. The frequency of topics varied among provider subspecialty. Geneticists' utterances were composed of “medical information” (27.8% [n = 3625]) and “noting previous statements” (16.9%, n = 2202). Urologists/gynecologists and endocrinologists were the most medically focused in their speech, talking about “medical information” more than a third of the time, 37.8% (n = 2202) and 34.9% (n = 2144), respectively. Urologists/gynecologists commented on “testing/treatment” 19.0% (n = 1106) of the time, while the topic was mentioned by endocrinologists with a proportion of 13.7% (n = 842). By comparison, psychologists spent significantly more time on “psychosocial information” (18.3% [n = 964]), with less time on “medical information” (16.6% [n = 874]). Families spent much less time on “medical information” (18.1% [n = 4216]) and more time on “psychosocial information” (7.8% [n = 1817]), which contrasted with the health care providers (except for psychologists).

Both health care providers and parents asked questions; overall, 8.4% of utterances were questions. Of all family utterances, 4.4% were questions. Health care providers asked more than double the percentage of questions asked by families (11.6%), although this varied by provider. Geneticists asked the most questions, accounting for 14.7% of their total utterance count, followed by psychologists (13.9%), endocrinologists (9.2%), and urologists/gynecologists (5.7%). Psychologists also asked the highest percentage of open-ended questions (33.8%), compared with endocrinologists (22.7%), geneticists (16.3%), and urologists/gynecologists (14.5%).

Questionnaire data

Collection of quantitative data derived from questionnaires administered to parents of affected children was less complete than anticipated and would not support sufficiently powered inferential data analyses. More specifically, of the questionnaires administered before a clinical decision was made, 47 (74.6%) caregivers completed the Decision-making Preference scale and 39 (61.9%) completed the GAD-7/PHQ-9. Questionnaires administered after decisions were made took place in the context of interviews outside the clinic setting. Response rates diminished for all questionnaires (Decision-making Preference, GAD-7/PHQ-9, Decisional Conflict, Relational Communication, Source Credibility, and the modified Trust in Providers) to include 21 (33.3%) respondents. Fortunately, we also assessed some of the data collected via questionnaire (eg, Relational Communication) using the SCCAP, which we used to code clinic interactions between parents and health care providers. Additional data analyses are ongoing.

Surgical decision-making

We qualitatively coded audio recordings of interactions between specialist health care providers and 61 caregivers of 31 patients that occurred during regular specialty clinic appointments at the 3 academic medical centers to distinguish the arguments in favor of and against surgery brought up by either parents or clinicians and to map the course of decision-making. In the 31 cases, clinicians or parents raised the possibility of surgery for 23 patients. Talking points mentioned by clinicians or parents at least once included the following: option of not doing surgery (83%), mention that surgery is not urgent (78%), surgery is cosmetic/elective (74%), surgery is controversial (65%), future interests of the child (65%), and consultation with outside sources (eg, peer support or advocacy organization; 61%). Under circumstances of ambivalence or uncertainty on the part of either the provider and/or caregiver, caregivers tended to selectively attend to details and discount warnings (eg, a case in which parents initially came to clinic expressing a strong desire for surgery; health care providers shared balanced arguments for and against surgery; parents repeated the arguments that favored surgery and did not repeating nor discuss the counterarguments); providers emphasized details in favor of the decision they considered most appropriate in the situation.

We did not observe substantive differences in the communications between caregivers and providers before vs after the introduction of the DST.

Phase 3—Identifying Opportunities and Barriers to DST Implementation

Actual DST use

Actual use of the DST among caregivers who were provided access to the DST varied across and within families. We tracked use online with a built-in mechanism that showed when users completed each of the decision-making exercises. Few caregivers used the web-based DST as designed: 16% of caregivers (representing 15% of children) reviewed all content and completed decision-making exercises; 44% (representing 54% of children) used it at least partially; and 40% (representing 31% of children) did not use it past initial logins at clinic or home. Usage and patient age were related (F2,22 = 25.1; P < .001) such that those who used it fully had older children (mean, 4.2 years; P < .001) than those who used it partially (mean, 0.9 years) and those who did not use it past initial login (mean, 0.2 years; P < .001); the difference between the latter 2 groups did not reach statistical significance (P < .13).

Three caregivers, who reported experiencing technological issues, requested and received printed/paper copies of relevant DST elements and are included in the counts above. Clinical and research coordinators inquired of other participants who had not logged in after a period of 1 to 2 weeks after being granted access if they would like printed copies; all declined. Aside from initial interactions when caregivers were first allowed access to the DST, few clinicians explicitly mentioned its use during clinic visits.

Dissemination and integration

We performed a content analysis of transcribed interviews and written feedback of health care provider and patient advocate stakeholders who had the opportunity to work with the DST outside the clinical setting. We categorized the content into discrete domains and topics of concern: positive/negative features of DST, changes needed, relevance, format, timing for use as well as an identification of the factors influencing use of the DST, and decision support and impact on the SDM process. Participants identified the following needs: (1) to refine the DST for it to be viewed as legitimate by all stakeholder groups; (2) to convince providers to alter their usual model of care by using the DST in their practice; and (3) to create effective systems for delivering decision support and adequate follow-up.

More specifically, regarding refinement, stakeholders noted some technological glitches and a general reading level that was considered too high for parents. However, these were not universal critiques; eg, while advocates mentioned reading level concerns, parents did not. Additionally, some disagreement about terminology (eg, intersex vs DSD) used in the DST remains unresolved as different stakeholder groups hold differing opinions about appropriate terms. Concerned about overwhelming families at long, multidisciplinary follow-up visits, health care providers preferred introducing the tool in clinic and encouraging families to further use it at home, on the condition that they had proper instructions for using the tool and wrote down their questions and concerns. In contrast, RAO representatives underlined the need for medical providers to go through the DST with families.

Discussion

Decisional Context

When children are born with DSD, parents and clinicians have many decisions to make on the child's behalf. Elective clinical management decisions (some irreversible) on behalf of newborns and young children with DSD are often made at times when parents feel intense emotional distress and struggle with learning about their child's complex medical condition. Anxiety-driven decisions often reflect limited knowledge of the range of options and inadequate weighing of the potential risks and benefits. Former patients and health care advocates complain that parents are provided inadequate information with which to make decisions for their child. Research corroborates these complaints: Parents often have not heard of DSD conditions before their child's birth and typically do not perceive that treatment choices exist; instead, many assume the diagnosis implies a single treatment path.

Project Results in Context

Decision aids and support tools are commonly found in other areas of adult health care62; this study represents the first attempt to create and pilot a DST for DSD. Based on findings from other areas of medicine where decision aids have been introduced, we anticipated that creation of a DST for DSD would help provide parents with adequate background information with which to make informed decisions that identify and incorporate personal and cultural beliefs and family values. The DST should provide a common framework and language for stakeholders who come from different backgrounds to communicate clearly with one another. Additionally, results related to the examining family-provider communication during clinic visits portion of the study highlight areas for future hypothesis generation and testing.

Implementation of Study Results

Regarding actual DST use among parents and health care providers during pilot testing, our results are in line with other research showing that fewer than expected parents were directed to use the web-based DST.63 Potential barriers noted by others include technical problems; clinicians' limited understanding of how patient DSTs could be helpful; clinician perception that SDM for DSD treatment was already commonplace; and external factors, such as efficiency targets and best-practice recommendations.63

More specifically, evidence that some clinicians did not fully grasp how a DST could be helpful was reflected in the comment “but we already do that” (ie, implying that all aspects of the DST—including education, values identification, and a formal weighing of family values and preferences in light of available evidence—is already routinely accomplished for each patient for each relevant decision) when introduced to the DST. This and the manner in which the DST was introduced may have contributed to the fewer than expected instances of clinicians recommending DST use during the context of clinic appointments. In contrast to much of the research literature in which decision aids and support tools are delivered to the intended audience (patients, families) in a highly structured manner, commonly involving the research team, the DST was administered by clinicians in a real-world environment under circumstances of usual care. Several of the clinicians who delivered the DST to caregivers had participated in its development; an assumption guiding rollout was that involvement in the creation and development of the DST—including decisions on how it would be used with caregivers—would translate to uptake by involved clinicians who would also model DST use to others. Clinicians not directly involved in DST development were introduced to the DST by clinician members of the research team via formal presentation and informal discussion before rollout. Messages were reinforced during preclinic meetings when the clinical team made decisions regarding which modules were most relevant for each family. However, clinicians received no specific training in how to translate this type of involvement into talking points with the family or consideration of how use of the DST might alter clinician workflow. Ultimately, as shown here and in other research,63 simply developing web-based interventions will not solve the problem of failing to incorporate DSTs into routine practice; rather, a broader implementation strategy will be required to integrate DSTs into routine practice.

Generalizability

DST Development

DSD comprises several medical conditions for which the “right” decisions about the best course of action is not always obvious and can, at times, become controversial. We sampled a diverse group of stakeholders to help identify components and provide feedback on the DST's development. Stakeholders included parents of affected children, patient resource and advocacy leaders and members, specialty (endocrinology, genetic counseling, genetics, gynecology, psychology, nursing, and urology/surgery) and PCPs, bioethicists, and others. It was intended, and was borne out in his project, that sampling this variety of stakeholders should result in a wide range of perspectives. As such, we included shared/cross-cutting perspectives across groups in the provisional DST (eg, educating families, supporting families, providing known pros and cons about procedures). We did not included in the DST design positions stated by any one stakeholder group that conflicted with all other stakeholder group perspectives; eg, declaration that genital surgery is tantamount to torture (see the “Subpopulation Considerations” section).

DST Development and Use

Families who were recruited in clinic were predominantly Caucasian, non-Hispanic. Except for phase 2, stage 1 caregivers who participated before DST development, participation was limited to those who had functional literacy in English, as we created the DST using the English language. It would be inappropriate to assume that concepts about DSD, its clinical management, and SDM would be understood in a similar fashion across all cultures. Further, each of the clinics whose clinicians extensively contributed to the development and/or use of the DST can be described as large regional medical centers whose catchment area extends greatly beyond the county (and even state) in which it is located. Each center was a member of the NIH-funded DSD-Translational Research Network38; ie, a network of clinical sites dedicated to setting standards of care for patients and families living with DSD. Each site offered the services of a multidisciplinary clinical team whose members—endocrinologist, geneticist, urogenital surgeon, and psychosocial support professionals—are trained in the specific care of DSD patients and their families. Member sites were also continuously engaged in research to assess and respond to the specific needs of those affected by DSD. To the extent that center characteristics may affect results, generalizability to other settings is not fully known.

Subpopulation Considerations

We did not design this study to compare DSD subpopulations based on the efficacy of DST delivery; nevertheless, development of this DST generated questions regarding differential effectiveness across the spectrum of specific conditions that fall under the DSD umbrella. DSD is an umbrella term for a myriad of conditions that affect the embryologic and fetal development of genital and reproductive structures—these are caused by a variety of identified and, as of yet, unidentified genetic variations, each with a range of body phenotypes. For every newly identified person with DSD, there may be multiple decision points to consider—most of which are challenging for 1 or more decision makers. Whereas gender assignment in people with congenital adrenal hyperplasia or proximal hypospadias may not be considered challenging by some, decisions such as whether to perform early genital surgery to normalize appearance and function are fraught with controversy within and across stakeholder communities. A specialist experienced in DSD care is likely to have developed, and will choose among, several schemas for managing cases based on what he or she views as defining characteristics of the patient (eg, genital appearance, function, personal understanding of the influence of prenatal hormone exposure on gender development). A new parent, however, will not have the same frame of reference as the experienced clinician. Even after karyotype and anatomic findings are shared, parents may continue to question the pros and cons of genetic testing, surgery and its timing, and talking to others (eg, extended family, friends, the child himself or herself) about the DSD. Parents will have questions: how much to say, to whom and at what time, and whether to contact patient support and advocacy organizations. These questions and decision points are the same across all DSD conditions and potentially have profound effects on family adjustment, parenting, and the emerging self-concept of the affected person. Additionally, former patient-advocates may have developed different schemas that point to different challenges and influence the context in which clinicians and parents or patients make decisions.

In the research for this study, we observed different stakeholder perspectives about DST content. These include the following: Geneticists wanted to include highly detailed information on genetic testing and causes of DSD as well as “carrier testing”; patient advocates encouraged limiting information to “just the basics,” so that parents could identify occasions when decision-making opportunities exist, the pros and cons of each option, and the known risks and benefits, without becoming overwhelmed; specialist health care providers urged that it be a stand-alone product that families could use largely on their own; while patient advocacy representatives called for its use in the clinical setting alongside the providers. Additionally, some patient advocates equated all elective surgical interventions on genital or reproductive structures to “torture.” Some of these advocates have worked with the UN High Commissioner for Human Rights and the UN Special Rapporteur on Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment to call for the “prohibition of surgery and treatment on the sex characteristics of minors without informed consent.”23 Further, some of these advocates called for the inclusion of this perspective in the DST. In contrast, other patient advocates and health care professionals opposed inclusion of this material out of concern that it would no longer allow for parental choice and decision-making on behalf of their child regarding surgery. Ultimately, how to settle these differences will require evidence that stems from systematic research in these areas.

An additional consideration for this type of research involves ensuring that expectations about the role of each stakeholder group are made explicit—particularly in light of the differing perspectives offered between stakeholder groups on genital surgery. In areas in which disagreement occurred, final decisions on what was included rested on the research team—and, ultimately, the PI—rather than any one stakeholder group. The structure and governance involved should be clear in the mind of the research team and shared with all prospective participating stakeholders so that individuals or organizations can opt in or out accordingly at the project's outset.

Study Limitations

The 3 participating medical sites were all members of the NIH-funded DSD-Translational Research Network. While that created ready-made pathways for cross-institutional research (eg, creating and piloting the DST), their use raises questions about the generalizability of results outside the network. Additionally, generalizability across cultures, particularly those in which English is not a primary language, requires further study.

Another limitation rests in how the DST was introduced (eg, the lack of formal training regarding talking points to be used with caregivers, anticipation of changes to provider workflow) and intended to be used; ie, presented to families in clinic for them to use, primarily, at home, with clinicians prepared to review progress and answer questions brought in by the family. While this was the preferred method of integration by health care providers, it differed from that of patient advocacy leaders, who, later in the project period, urged its use in clinic in addition to home use. How results may have differed had another approach been adopted from the outset is open to question. Of note, while all caregiver in-clinic activities progressed without changes in participation (eg, no dropouts owing to being “too busy,” few unreturned questionnaires), activities outside of clinic (including postdecision/postclinic recording interview participation and actual DST usage) were poorly attended, with participants citing “too busy” as a reason not to participate. Additionally, although we asked questions about usability and user acceptance during focus groups and stakeholder interviews, we did not conduct formal usability testing; it is possible that we could have addressed some technological issues affecting use had formal usability testing been a component of the project.64 Finally, while few clinicians explicitly mentioned its use during clinic appointments, information about which sections were (or were not) completed by participants was communicated to each patient's clinical providers by the research coordinator before clinic visits. It is possible that this knowledge was incorporated into their presentation; however, this hypothesis was not formally tested.

Future Research

Integrating the DST Into the Clinical Setting

The Michigan-based research team has continued to refine and revise the DST based on stakeholder feedback, including transitioning it from a web-based tool to a modular workbook that can be tailored to family needs specific to the affected patient's characteristics. Additional research on methods of integrating its use into usual care will occur, first at the University of Michigan and then at sites participating in the DSD-TRN (the DSD-TRN currently includes Michigan, Seattle, and 10 other sites).38

We will use the Plan-Do-Study-Act (PDSA) cycle65 as the framework for further development. Described as “the scientific method, used for action-oriented learning,” a complete PDSA cycle involves monitoring performance, implementing operational changes, reviewing the effect of each change, and adjusting operations accordingly; this cycle is intended to be repeated until satisfactory results are found.65 Viewed from within the PDSA framework, the piloting testing described in this report comprises the first elements in the cycle.

Expectations Regarding Research Across Stakeholder Groups

Given potential differences in how academic medical researchers and nonresearchers view research, its purpose, and its pace, we recommend more attention to better defining attitudes and expectations about the role and pace of research across stakeholder groups.

Defining success

Differences exist in how stakeholder groups define successful outcomes for treatment of DSD. Future research should systematically gather information on how members of each stakeholder group define “successful outcomes” in DSD and the “best” way to achieve these outcomes. Additionally, research is planned to learn what tradeoffs stakeholders would make when 1 outcome conflicts with another (eg, managing aspects of privacy vs disclosure, patient involvement in decision-making about surgery vs conducting surgery at a young age so the child has no “bad” memories of the surgical events).

Conclusions

Despite calls for an interactive web-based DST that promotes SDM, only a minority of families fully used the tool, either at home or in clinic, and few clinicians mentioned its use within the context of clinic appointments. Our results highlight considerable structural and attitudinal challenges to overcoming issues that underpin the status quo in decision-making in routine DSD health care. Conceptualizing what has been accomplished in this project in terms of the 4 steps of the PDSA cycle,65 we accomplished the Plan—step 1—to add structure to the SDM process through introduction of a web-based DST, along with a strategy for collecting data related to its meaningful use. We completed the Do—step 2—by piloting the DST in a pilot sample. Our analysis of data and study findings constituted Study—step 3. Based on lessons learned, ACT—step 4—began by refining the presentation of the DSD by creating a workbook companion to the web-based DST. Lessons learned from the pilot are also leading us to create an educational resource for specialist providers before the health care team's use of the DST in usual care.

This project has taught us that meaningful assessment of the value of the DST in SDM requires that its integration within usual care—a substantial deviation from existing workflows—be embedded within accepted mechanisms for quality improvement in health care; eg, the PDSA cycle. Ultimately, the value of the DST needs to be evaluated by assessing the extent of parental engagement in and quality of parental decision-making on behalf of their newborn or young DSD-affected child and the reduction or absence of decisional regret in the face of continuous medical discoveries, modifications of interventions, and potential shifts in population understanding of and attitudes toward DSD.

References

1.
Lee PA, Houk CP, Ahmed SF, Hughes IA, International Consensus Conference on Intersex organized by the Lawson Wilkins Pediatric Endocrine Society and the European Society for Paediatric Endocrinology. Consensus statement on management of intersex disorders. International Consensus Conference on Intersex. Pediatrics. 2006;118(2):e488-e500. [PubMed: 16882788]
2.
Pasterski V, Mastroyannopoulou K, Wright D, Zucker J, Hughes I. Predictors of posttraumatic stress in parents of children diagnosed with a disorder of sex development. Arch Sex Behav. 2014;43(2):369-375. [PubMed: 24085468]
3.
Tamar-Mattis A, Baratz A, Baratz Dalke K, Karkazis K. Emotionally and cognitively informed consent for clinical care for differences of sex development. Psychol Sex. 2013;5(1):44-55.
4.
Duguid A, Morrison S, Robertson A, et al. The psychological impact of genital anomalies on the parents of affected children. Acta Pædiatr. 2007;96(3):348-352. [PubMed: 17407454]
5.
Dreger AD. Hermaphrodites and the Medical Invention of Sex. Harvard University Press; 1998.
6.
Karkazis K. Fixing Sex: Intersex, Medical Authority, and Lived Experience. Duke University Press; 2008.
7.
Feder EK. Making Sense of Intersex: Changing Ethical Perspectives in Biomedicine. Indiana University Press; 2014.
8.
Chuang J, Vallerie A, Breech L, et al. Complexities of gender assignment in 17β-hydroxysteroid-hydroxysteroid dehydrogenase type 3 deficiency: is there a role for early orchiectomy? Int J Pediatr Endocrinol. 2013;(1):15. [PMC free article: PMC3847283] [PubMed: 24025597]
9.
Maimoun L, Philibert P, Cammas B, et al. Phenotypical, biological, and molecular heterogeneity of 5{alpha}-reductase deficiency: an extensive international experience of 55 patients. J Clin Endocrinol Metab. 2011;96(2):296-307. [PubMed: 21147889]
10.
Jackson C, Cheater FM, Reid I. A systematic review of decision support needs of parents making child health decisions. Health Expect. 2008;11(3):232-251. [PMC free article: PMC5060453] [PubMed: 18816320]
11.
Lipstein EA, Dodds CM, Britto MT. Real life clinic visits do not match the ideals of shared decision making. J Pediatr. 2014;165(1):178-183. [PMC free article: PMC4106460] [PubMed: 24795203]
12.
Crissman HP, Warner L, Gardner M, et al. Children with disorders of sex development: a qualitative study of early parental experience. Int J Pediatr Endocrinol. 2011;2011(1):10. [PMC free article: PMC3223129] [PubMed: 21992519]
13.
Sandberg DE, Gardner M, Cohen-Kettenis PT. Psychological aspects of the treatment of patients with disorders of sex development. Semin Reprod Med. 2012;30(5):443-452. [PMC free article: PMC3687360] [PubMed: 23044882]
14.
Baratz AB, Sharp MK, Sandberg DE. Disorders of sex development peer support. In: Hiort O, Ahmed S, eds. Understanding Differences and Disorders of Sex Development (DSD). Endocrine Development. Karger; 2014:99-112. [PubMed: 25247648]
15.
Kon AA. Ethical issues in decision-making for infants with disorders of sex development. Horm Metab Res. 2015;47(5):340-343. [PubMed: 25970711]
16.
Karkazis K, Tamar-Mattis A, Kon AA. Genital surgery for disorders of sex development: implementing a shared decision-making approach. J Pediatr Endocrinol Metab. 2010;23(8):789-805. [PubMed: 21073122]
17.
Tamar-Mattis A, Baratz A, Baratz Dalke K, Karkazis K. Emotionally and cognitively informed consent for clinical care for differences of sex development. Psychol Sex. 2013;5(1):44-55.
18.
Saussar L. Lawyers prepare for medical malpractice trial in rare ‘intersex’ lawsuit. The Post and Courier. May 25, 2015. https://www​.postandcourier​.com/archives/lawyers-prepare-for-medical-malpractice-trial-in-rare-intersex-lawsuit​/article_8100bb96-7d0e-57e9-a76a-e1bd14763e3b.html
19.
20.
Bennett-Smith M. Mark and Pam Crawford, parents of intersex child, sue South Carolina for sex assignment surgery [VIDEO]. Huffington Post. May 15, 2013. https://www​.huffpost​.com/entry/mark-pam-crawford-intersex-child_n_3280353
21.
Greenfield C. Should we “fix” intersex children? The Atlantic. July 8, 2014. https://www​.theatlantic​.com/health/archive​/2014/07/should-we-fix-intersex-children/373536/
22.
Groundbreaking SPLC lawsuit accuses South Carolina, doctors and hospitals of unnecessary surgery on infant. Southern Poverty Law Center. Updated May 13, 2013. Accessed December 6, 2015.
23.
New UN intersex fact sheet. Advocates for Informed Choice; 2015. Accessed December 5, 2015.
24.
Funding anesthesia research to ensure pediatric safety. SmartTots. Published 2015. Accessed December 6, 2015. http://smarttots​.org/
25.
Siminoff LA, Sandberg DE. Promoting shared decision making in disorders of sex development (DSD): decision aids and support tools. Horm Metab Res. 2015;47(5):335-339. [PubMed: 25853896]
26.
Politi MC, Dizon DS, Frosch DL, Kuzemchak MD, Stiggelbout AM. Importance of clarifying patients' desired role in shared decision making. Br Med J. 2013;347:f7066. [PubMed: 24297974]
27.
Charles C, Gafni A, Whelan T. Decision-making in the physician-patient encounter: revisiting the shared treatment decision-making model. Soc Sci Med. 1999;49(5):651-661. [PubMed: 10452420]
28.
Charles C, Gafni A, Whelan T. Shared decision-making in the medical encounter: what does it mean? (or it takes at least two to tango). Soc Sci Med. 1997;44(5):681-692. [PubMed: 9032835]
29.
Frader J, Alderson P, Asch A, et al. Health care professionals and intersex conditions. Arch Pediatr Adolesc Med. 2004;158(5):426-428. [PubMed: 15123472]
30.
Chase C. What is the agenda of the intersex patient advocacy movement? Endocrinologist. 2003;13(3):240-242.
31.
Greenberg JA. Legal, ethical, and human rights considerations for physicians treating children with atypical or ambiguous genitalia. Semin Perinatol. 2017;41(4):252-255. [PubMed: 28478089]
32.
U.S.: harmful surgery on intersex children: medically unnecessary operations risk lifelong suffering. News release. Human Rights Watch; July 25, 2017. https://www​.hrw.org/news​/2017/07/25/us-harmful-surgery-intersex-children
33.
Commissioner for Human Rights. Human Rights and Intersex People. Council of Europe. Published 2015. Accessed August 21, 2016. https://rm​.coe.int/human-rights-and-intersex-people-issue-paper-published-by-the-council-/16806da5d4
34.
United Nations Human Rights Council, Méndez JE. Report of the Special Rapporteur on Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment. Published February 1, 2013. Accessed February 7, 2013. https://www.ohchr.org/documents/hrbodies/hrcouncil/regularsession/session22/a.hrc.22.53_english.pdf
35.
Wyatt KD, List B, Brinkman WB, et al. Shared decision making in pediatrics: a systematic review and meta-analysis. Acad Pediatr. 2015;15(6):573-583. [PubMed: 25983006]
36.
Richardson EJ, Scott FP, McLennan AC. Sex discordance identification following non-invasive prenatal testing. Prenat Diagn. 2017;37(13):1298-1304. [PubMed: 29131362]
37.
Accord Alliance. Accessed May 17, 2014. http://www​.accordalliance.org/
38.
About the DSD-TRN. Disorders/Differences of Sex Development-Translational Research Network. Accessed June 27, 2017. https://dsdtrn​.org/for-patients-families/about-dsdtrn/
39.
O'Connor AM. Ottawa decision support framework to address decisional conflict. The Ottawa Hospital; 2006. https://decisionaid​.ohri.ca/odsf.html [PubMed: 16855126]
40.
O'Connor AM, Stacey D, Jacobsen MJ. Ottawa personal decision guide. The Ottawa Hospital; 2015. https://decisionaid​.ohri​.ca/docs/das/OPDG.pdf
41.
Elwyn G, O'Connor AM, Stacey D, et al; International Patient Decision Aids Standards (IPDAS) Collaboration. Developing a quality criteria framework for patient decision aids: online international Delphi consensus process. Br Med J. 2006;333(7565):417. [PMC free article: PMC1553508] [PubMed: 16908462]
42.
O'Connor A, Jacobsen M. Workbook on developing and evaluating patient decision aids. The Ottawa Hospital; 2003. http://decisionaid​.ohri​.ca/docs/develop/Develop_DA.pdf
43.
O'Connor AM, Rostom A, Fiset V, et al. Decision aids for patients facing health treatment or screening decisions: systematic review. Br Med J. 1999;319(7212):731-734. [PMC free article: PMC28223] [PubMed: 10487995]
44.
Birt L, Scott S, Cavers D, Campbell C, Walter F. Member checking: a tool to enhance trustworthiness of merely a nod to validation? Qual Health Res. 2016;26:1802-1811. [PubMed: 27340178]
45.
Volk, R. Beyond readability formulas: writing clear and effective patient-centered educational materials. Michigan Medical Headlines. University of Michigan; 2016.
46.
Nancekivell S. Plain language alternatives for patient information and consent materials. Published December 5, 2008. https://www.orpca.org/files/Plain_Language_Alternatives_for_Patient_Information_and_Consent_Materials.pdf
47.
Centers for Disease Control and Prevention. Plain language thesaurus for health communications. 2007. https://www​.cdc.gov/healthliteracy​/developmaterials​/plainlanguage.html
48.
Zucker KJ. Re: sexual behaviors, sexual orientation and gender identity in adult intersexuals: a pilot study. J Urol. 2002;168(4 Pt 1):1507-1508. doi:10.1016/s0022-5347(05)64501-6 [PubMed: 12352443] [CrossRef]
49.
O'Connor AM. Validation of a decisional conflict scale. Med Decis Making. 1995;15(1):25-30. [PubMed: 7898294]
50.
Degner LF, Kristjanson LJ, Bowman D. Information needs and decisional preferences of women with breast cancer. JAMA. 1997;277(18):1485-1492. [PubMed: 9145723]
51.
Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. JAMA. 1999;282(18):1737-1744. [PubMed: 10568646]
52.
Burgoon JK, Hale JL. Validation and measurement of the fundamental themes of relational communication. Commun Monogr. 1987;54:19-41.
53.
Berlo DK, Lemert JB, Mertz RJ. Dimensions for evaluating the acceptability of message sources. Public Opin Q. 1969;33(4):563-576.
54.
Anderson LA, Dedrick RF. Development of the Trust in Physician scale: a measure to assess interpersonal trust in patient-physician relationships. Psychol Rep. 1990;67(3 Pt 2):1091-1100. [PubMed: 2084735]
55.
Moseley KL, Clark SJ, et al. Parents' trust in their child's physician: using an adapted Trust in Physician scale. Ambul Pediatr. 2006;6(1):58-61. [PubMed: 16443185]
56.
Siminoff LA, Step MM. A comprehensive observational coding scheme for analyzing instrumental, affective, and relational communication in health care contexts. J Health Commun. 2011;16(2):178-197. [PMC free article: PMC3147015] [PubMed: 21213170]
57.
Tavory I, Timmermans S. Abductive Analysis: Theorizing Qualitative Research. University of Chicago Press; 2014.
58.
Musen MA, Middleton B, Greenes RA. Clinical decision-support systems. In: Shortliffe EH, Cimino JJ, eds. Biomedical Informatics. Springer; 2014:643-674.
59.
Creswell JW. Qualitative Inquiry and Research Design Choosing among Five Traditions. Sage Publications; 1998.
60.
Charise A, Witteman H, Whyte S, et al. Questioning context: a set of interdisciplinary questions for investigating contextual factors affecting health decision making. Health Expect. 2011;14(2):115-132. [PMC free article: PMC5060568] [PubMed: 21029277]
61.
Lau AYS, Siek KA, Fernandez-Luque L, et al. The role of social media for patients and consumer health. Contribution of the IMIA Consumer Health Informatics Working Group. Yearb Med Inform. 2011;6:131-138. [PubMed: 21938338]
62.
Stacey D, Bennett CL, Barry MJ, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev. 2017;(4):CD001431. doi:1002/14651858.CD001431.pub5 [PMC free article: PMC6478132] [PubMed: 28402085]
63.
Elwyn G, Rix A, Hold T, Jones D. Why do clinicians not refer patients to online decision support tools? Interviews with front line clinics in the NHS. BMJ Open. 2012;2:e001530. doi:10.1136/bmjopen-2012-001530 [PMC free article: PMC3532981] [PubMed: 23204075] [CrossRef]
64.
65.
Plan-Do-Study-Act (PDSA) cycle. Agency for Healthcare Research and Quality website. Updated April 10, 2013. Accessed December 7, 2015. https://innovations​.ahrq​.gov/qualitytools​/plan-do-study-act-pdsa-cycle

Acknowledgment

Research reported in this report was [partially] funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (#1360). Further information available at: https://www.pcori.org/research-results/2012/helping-caregivers-children-differencesdisorders-sex-development-decide

Original Project Title: Decision Support for Parents Receiving Genetic Information about Child's Rare Disease
PCORI ID: 1360
ClinicalTrials.gov ID: NCT01875640

Suggested citation:

Sandberg DE, Gardner M, Callens N, et al. (2019). Helping Caregivers of Children with Differences/Disorders of Sex Development Decide on Treatment. Patient-Centered Outcomes Research Institute (PCORI). https://doi.org/10.25302/11.2019.CER.1360

Disclaimer

The [views, statements, opinions] presented in this report are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee.

Copyright © 2019. University of Michigan. All Rights Reserved.

This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License which permits noncommercial use and distribution provided the original author(s) and source are credited. (See https://creativecommons.org/licenses/by-nc-nd/4.0/

Bookshelf ID: NBK599377PMID: 38252770DOI: 10.25302/11.2019.CER.1360

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (3.5M)

Other titles in this collection

Related information

  • PMC
    PubMed Central citations
  • PubMed
    Links to PubMed

Similar articles in PubMed

See reviews...See all...

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...