Validation of a Parent-Reported Experience Measure of Integrated Care

Boston Children’s Hospital (BCH) Institutional Review Board approved all parts of this study (P00008144; P00007912). The Lucile Packard Children’s Hospital Institutional Review Board approved the study protocol through a reliance agreement.

A pilot survey of parent-reported experiences with pediatric care integration was developed with substantial input from parents. Between July and December 2013, six focus groups with a total of 36 parents and 4 additional semi-structured interviews were conducted with parents of children receiving care from multiple medical and nonmedical providers. Interviews were conducted in English except for 1 interview, which was conducted in Spanish using a language interpreter. The families were recruited through the BCH Department of Psychiatry, BCH Primary Care Center, and private, community-based, pediatric primary care practices. The Massachusetts Federation for Children with Special Needs also recruited families by sending out study information to their listserv. Families were eligible to participate if they reported having a child who, in the past 12 months, had had at least 1 primary care appointment and at least 2 specialty appointments (with ≥1 specialist), defined broadly as medical, surgical, behavioral specialties, and allied health professions. All participants were parents of CYSHCN with various medical, behavioral health, and educational needs; various institutional affiliations of their primary and specialty care providers; and varying degrees of dependence on allied health and durable medical equipment providers (Supplemental Table 5). Participants were asked to describe what integrated care meant to them, how they would characterize care that is integrated across all care team members, and how care should be integrated from the perspective of families. Thematic analyses of focus group transcripts resulted in a framework of family-centered care integration (Fig 1).

Families reported that, essentially, integrated care is an approach that treats their children and youth “holistically” and which includes 3 domains: team-based care; connection to life/community; and the future (care planning). Team-based care encompasses elements of team composition, methods of efficient communication, and purposeful information sharing. The second domain, linkage to the broader community, was described by families as connection to aspects of “life” beyond medical care. Families expressed that assessment of both their strengths and needs is essential to the process by which appropriate resources are identified for them. Assisting families in accessing indicated medical and nonmedical resources is vital to achieving optimal outcomes. The third domain, the future, encompassed a formal, inclusive process of care planning. Families described the need for time-bound goals, taking into account the multiplicity of tactics and skills necessary to achieve both short- and long-term outcomes. Families and care team members must collaborate to prioritize goals and ensure that the necessary resources can be secured to achieve those goals. Underlying each of these elements is an expectation of accountability for organizing and coordinating care, with intentionality to improve care integration efforts and outcomes.

A first draft of the pilot survey was based on the Patient Perceptions of Integrated Care Survey, the data collected in the focus groups, and a literature review of existing parent-reported surveys on care experiences as well as measures of CC and care integration.^6,–12 In addition, an expert advisory panel provided input on the structure and content of the survey during the initial survey development. The 17 panelists were selected based on their experience and knowledge related to family advocacy, care of CYSHCN, health policy, quality measurement, CC, care integration, and survey methodology. Representatives of family organizations were recruited to ensure that the perspectives of families were included during all phases of this project.

Between December 2013 and February 2014, eight cognitive interviews were conducted to assess terminology and sentence structure, and to identify gaps and vulnerabilities in respondents’ ability to retrieve experiences from memory. Survey questions were revised based on the feedback.

The final pilot survey contained 95 questions, organized into 8 thematic sections (Supplemental Table 6). Responses for the experience items were generally 6-point Likert scales with the response options “never,” “rarely,” “sometimes,” “usually,” “almost always,” and “always.”

Participants for pilot testing of the survey were recruited between July 2014 and June 2015. Recruitment procedures and eligibility criteria for the pilot test were the same as for the focus groups. In-person recruitment took place at 4 sites: BCH; Lucile Packard Children’s Hospital, Palo Alto, California; and at state and regional meetings of the Massachusetts Federation for Children with Special Needs in 2014 and 2015. Four collaborating organizations (the Massachusetts Federation for Children with Special Needs, the Parent/Professional Advocacy League of Massachusetts, Westwood-Mansfield Pediatrics [a community-based primary care setting], and the Crotched Mountain School, Greenfield, New Hampshire) electronically sent recruitment leaflets to their families of CYSHCN.

All participants in the pilot study were invited to complete the PICS at least once. The 259 participants recruited before March 2015 were also asked to complete the PICS a second time ∼2 weeks after completing the first survey to assess test–retest reliability. BCH study staff sent out all surveys and conducted all follow-up activities.

PICS were administered either by paper through mail or electronically through Qualtrics (Provo, UT),¹³ depending on participant preference. Each participant received up to 3 reminder e-mails and/or 2 reminder telephone calls if they had not completed the survey. Monetary stipends were sent for each completed survey.

All statistical analyses were performed by using Stata version 14.0 (Stata Corp, College Station, TX).¹³ Item-level descriptive statistics were evaluated by using frequencies, percentages, and univariate measures of location and variability. Exploratory factor analysis (EFA) was used to determine which of the experience items should be included in the final questionnaire and the underlying measurement domains each item represented. Psychometric properties of the final questionnaire were evaluated by using tests of construct validity, internal consistency, and test–retest reliability.^14,15 

The experience items in the EFA for the construction of the core instrument needed to have a minimum sample size to allow for the factor analytic statistics to be estimated. Because answers to a number of items, including experience questions relating to school, transition/transfer to adult care, and care planning, were dependent on applicability of the screening questions, only experience items that were reportedly applicable to at least two thirds of the respondents were included in the EFA. Furthermore, items with >10% of respondents choosing either “don’t know” or “not applicable” were also excluded from the factor analysis. However, “don’t know” responses of experience items for which this option represented a lack of awareness of communication or information sharing were recoded as the lowest Likert scale response option. The justification for this choice was that the focus groups described awareness of these activities of communication and information sharing as being important to family experience of care integration. The 26 remaining items were then eligible to be included in the EFA.

There were no significant differences between respondents using electronic versus paper survey results (Supplemental Table 7).

Principal factor analysis with varimax rotation was used to explore the structure underlying the 26 experience items. Factor loadings guided which items measured the same underlying construct. The inclusion or exclusion of an item in a construct was determined iteratively by examining factor loadings (>0.3) and Cronbach’s α to identify redundant items or items that did not sufficiently measure the same underlying construct.

The composite score calculation method followed the proportional scoring method used for Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys. Detailed instructions can be found in the Supplemental Materials “PICS Composite Score Calculation.”

Internal consistency of the items within each measurement domain was tested by using Cronbach’s α. Values of α ≥0.7 were acceptable. For demonstration of discriminant validity, pairwise Spearman rank correlation coefficients between each item, their own composite, and all other composites were examined. Evidence of internal consistency between an item and its own composite scale was defined as a correlation coefficient ≥0.40. Correlations between an item and the other composite scales should, on average, be lower than between the item and its own composite scale.

For test–retest reliability, a weighted Cohen’s κ was used as an indicator for agreement between the individual items assessed by the same respondent measured at the 2 time points. Values >0.80 represent almost perfect agreement, those between 0.61 and 0.80 substantial agreement, values between 0.41 and 0.60 moderate agreement, and those <0.41 fair agreement.¹⁶ 

Construct validity tests were conducted by using known group comparisons. Composite measures should be able to discriminate between groups that are known to be different with regard to at least some of the constructs measured by using the survey tool. Children with characteristics that predict increased health care needs and usage are more likely to exhibit fragmented care² and therefore should have, on average, lower scores on the constructs. We used t tests adjusting for unequal variances to compare the mean composite scores between the identified groups.

In total, 442 parents and guardians agreed to receive the survey. The majority preferred to receive the survey electronically (n = 326 [73.8%]) compared with mail (n = 116 [26.2%]). Of those sent, 255 surveys were returned, for a 57.7% response rate; responses were higher for Web mode than for mail administration (electronic: n = 198 [60.7%]; mail: n = 57 [49.6%]; P = .04).

The first 259 participants were also invited to complete the PICS a second time after having returned the first survey. From this group of participants, 145 returned the first survey (56.6%) and of those, 81.3% returned the second survey (n = 118). There were, however, no significant differences between how participants returned the second survey (electronic: 84 of 103 [81.6%]; mail: 34 of 42 [81.0%]; P = .93).

Table 1 displays the demographic characteristics of the respondents, households, child, and the child’s health care needs. The majority of the respondents were female (92.1%) and had at least a 4-year college degree (63.7%). Nearly all children were reported to have a primary care provider (96.5%). Two-thirds of children were seen by medical or surgical specialists (65.1%) and/or allied health professionals, such as occupational, physical, and speech therapists (65.5%). Approximately one-half of the children used school care providers, such as school nurse or school counselor, and/or behavioral health care providers (53.3% and 45.1%, respectively). During PICS development, families reported that their experience of patient- and family-centered care integration was critically dependent on the designation of all professionals as “care providers” who offer services to their children. This label includes professionals from multiple disciplines including, but not limited to, medical, surgical, behavioral health, allied health, and educational providers. Therefore, the instrument encouraged families to define care provider as anyone whom they perceived to have delivered services to their child in the previous 12 months, with the expectation of having an impact on the child’s health.

Care teams ranged from 2 to 5 care providers (43.9%) or 6 to 10 care providers (36.0%). Nearly two-thirds of the families reported their providers were part of the same health system (62.0%). Respondent, household, and child characteristics did not differ across the recruitment sites (Supplemental Table 8). Significant differences were, however, found with regard to the size of the health care teams, the types of health care providers caring for the child, the type of medical conditions, and the use of services such as counseling, medical equipment, special therapy, and special education services. There were no significant differences in experience items or the composite mean scores as a function of survey administration mode (Supplemental Table 7).

A 5-factor model was chosen as the best analytic solution based on the scree plot of the 19 experience items, the factor loadings, parallel analysis (Supplemental Fig 2), and conceptual knowledge from the focus groups (Supplemental Tables 14-17 for models with factors 1–4). Table 2 displays the descriptive statistics for each item according to constructs. Cronbach’s α (Table 3) and the factor loadings excluded 7 of the 26 initial items from the core instrument. The factor loadings, the uniqueness of each item, and the proportion of the common variance of the item not associated with the factors, are shown in Supplemental Table 9. The 5 constructs underlying the 19-item core instrument are:

1. Access: access to care (2 items);

2. Communication: communication between care team members and parents/guardians (4 items);

3. Family impact: assessment and remediation of the impact the child’s health care needs have on the family (5 items);

4. Care goal creation: creation of short- and long-term care goals (2 items); and

5. Team functioning: team functioning and quality of the care team (6 items).

Except for 2 items in the “family impact” domain, all items showed factor loadings ≥0.5 on their respective factor. The Flesch-Kincaid Reading Ease Score of the final core instrument, the demographic background questions, and the health care characteristics and utilization questions was calculated to be 70.5 or a grade level of 8.6.¹⁷ 

Cronbach’s α values were close to or >0.7 for 4 of the 5 constructs (Table 3). Internal consistency was also established through strong correlation coefficients with values ≥0.4 between items and their own composite for 4 of the 5 constructs. Three family impact items showed moderate or weak correlation coefficients (≤0.33). However, the range and mean of the correlation coefficients between items and the other composite scales were always lower than those between the items and their own composite scale, demonstrating strong discriminant validity.

With regard to test–retest reliability, 10 items agreed substantially between the 2 survey administrations, with κ values >0.60. The other 9 items showed moderate agreement, with κ values between 0.41 and 0.60 (Table 3).¹⁶ 

Respondents were expected to have, on average, lower mean composite scores if they had the following: a child with increased health care needs because of ≥2 types of medical conditions; those with higher health care usage such as services from allied health professionals, counseling, medical equipment, and additional services; those with >2 health care providers or >1 medical specialist; those reported by parents to have needed more care than usual; and those reported by parents to have changing care needs. The distribution of the composite scores for each construct is shown in Table 4. Supplemental Table 10 presents the composite-to-composite correlation.

Construct validity was established by assessing the differences in mean composite scores between groups known to be different with regard to at least 1 of the 5 constructs. Questions from the PICS about health care needs and usage were used to differentiate the respondents into groups. If a child’s health care providers all have access to the same medical record, this access should, in theory, facilitate information-sharing, CC, and integration, and therefore yield, on average, higher composite scores. Supplemental Table 11 generally confirms these expectations. Nearly all health care needs and usage indicators show lower composite scores for those indicating higher needs or usage. Although the differences were significant for the composite scales access and team functioning, only about one-half of the indicators for the composite scales of communication and family impact reached statistical significance. However, differences in the mean scores were directionally as expected. If all health care providers had access to the same medical records, composite scores were, as expected, significantly higher for these 4 domains. For the composite scale of care goal creation, none of the groups exhibited significant differences, and the direction of the composite score means was not always as expected. Having services from allied health professionals, medical equipment, and using >2 health care providers showed increased mean scores. Parents of children using behavioral health care showed significantly lower mean scores on all composites and most individual items (Supplemental Table 12). Care received from providers within 1 health system was rated significantly higher with regard to 4 composites and most of the related items. The only composite showing no difference between the health care systems was care goal creation. Insurance type did not show any significant differences for composite and most individual item ratings.

Although frameworks for care coordination^1,3,18 and care integration⁴ have been described, assessing care integration as a patient- or family-reported experience measure has yet to be adopted as a critical outcome of delivery systems taking on accountability for both quality and cost.^19,20 

Due to the number of providers involved in their care, CYSHCN are at risk for fragmented care. The results of the family focus groups in the present study shed light on key aspects of care integration for this population and vulnerabilities that may result in poor integration. Focus groups reported that families themselves were most likely to be the care integrator for children and youth with multiple chronic conditions because these children often do not fit within the purview of any single, traditional clinical care model. In addition, families whose children have significant behavioral health needs, as well as those with unknown diagnoses, were often left on their own to integrate care across providers.

The PICS offers an instrument to complement both the Family Experience with Coordination of Care Survey measure (a tool for assessing families’ experience of care coordination for their children with medical complexity)²¹ and the Child CAHPS surveys.²² The Family Experience with Coordination of Care Survey measures family experience of CC at a tactical level, such as in response to providing access to a care coordinator or the availability of a care plan. The Child CAHPS surveys are important measures of care experiences received by children, focusing on interactions with specific care providers but not necessarily how the providers integrate their efforts in addressing the holistic needs of the child and family. In contrast, the PICS assesses a family’s experience of care integration across a team of providers, irrespective of institutional affiliation, the discipline of the provider, the type of intervention, or the location of care. As such, the PICS may prove valuable for those stakeholders who are approaching care redesign more broadly. Although the PICS can measure the impact of the implementation of a care plan, it may also be effective in assessing the family experience outcome of a broad array of interventions, such as the implementation of innovative collaborative care models. These models aim to transform how primary and specialty care providers collaborate, and early experience has been favorable with conditions such as constipation,²³ behavioral health,²⁴ and obesity.²⁵ Furthermore, because the PICS enables the family to identify all members of a multidisciplinary team, it can be used to evaluate care integration between medical, educational, behavioral, and social service providers.

The PICS can be used to assess family experience of care integration within existing care delivery models and systems and therefore guide quality improvement efforts in those settings. Given that medically complex children are increasingly identified as a key population for care management, the PICS can be used to assess baseline integration experience, as new care models are designed and implemented. Due to its ability to assess integration across disciplines (eg, medical, behavioral, educational, family support), the PICS can be used to identify gaps in care delivery. As delivery systems begin to take on accountability for populations with medical and social determinant risk factors, the PICS might serve a critical role in evaluating broader care integration needs.

The PICS should be viewed in light of its limitations. Although we received family input into the design of the PICS instrument from families in California and Massachusetts, to date it has only been validated for families living in the Boston area. In addition, our family focus groups suggested a number of tactics and tools that they felt would improve their perception of care integration, such as care planning. However, in our pilot survey, the number of families who reported having experience with care planning was sufficiently small that we were not able to include them in the EFA. Despite this limitation, we have chosen to honor the wisdom of our family collaborators by retaining items that families felt would be of high potential value as additional modules, even if not yet mapping to a psychometric domain. The final PICS includes 26 rating items and 29 demographic and background questions that can be selectively included depending on how much information is already known about the respondent’s child. Our expectation is that further implementation of the PICS will enable us to gather the necessary data to expand the validated core set of questions. We have translated the PICS into Spanish, but its psychometric properties have not yet been determined.

Future research should focus on assessing the predictive validity of PICS that could not be tested in this study because collecting outcome measures beyond parent-reported experience of care integration was beyond the scope of the project. Correlational analyses with other existing instruments measuring the quality of care experiences should also be undertaken to assess concurrent validity. The inclusion of additional items that could not be included in the psychometric analyses in this study due to low prevalence in traditional health care, such as existence of a written care plan, might also further improve the psychometric characteristics of PICS. Administration of PICS in other settings and subsequent confirmatory factor analysis is therefore recommended. Given the nascent state of care integration, PICS is expected to evolve to meet the performance evaluation needs of a transforming health care ecosystem, as family experience of care integration justifiably emerges as a normative outcome.

The PICS is a validated instrument designed to measure parent-reported experience of care integration for children and youth who, in the past 12 months, had 1 primary care encounter and at least 2 specialty encounters. It was designed to broadly assess family experience across the care continuum. As a result of its strong psychometric properties, the PICS may be used to inform quality improvement efforts within and across delivery systems and communities, reflecting families’ experience of care among medical and nonmedical providers, irrespective of organizational affiliations. Broad adoption of the PICS by delivery systems will depend on additional research to assess implementation experience. Using this measure of authentic, family-reported care integration may serve to advance performance measurement of the patient and family experience domain of the Institute for Healthcare Improvement's Triple Aim.²⁶ 

BCH

Boston Children’s Hospital

CAHPS

Consumer Assessment of Healthcare Providers and Systems

CYSHCN

children and youth with special health care needs

EFA

exploratory factor analysis

CC

care coordination

PICS

Pediatric Integrated Care Survey

The authors are grateful to Dr Edward Schor, Senior Vice President at the Lucile Packard Foundation for Children’s Health, for his support, inspiration, and guidance throughout this project. They acknowledge the contributions of the expert advisors, family and community groups, health care professionals, and families who have contributed to the development and validation of the PICS. The authors especially thank Ms Cathy Hickey and Mr Rich Robison at the Massachusetts Federation for Children with Special Needs for their collaboration throughout this project.