Sepsis Prediction in Hospitalized Children: Model Development and Validation

This study describes development of a single-center prediction model and prospective validation of that model. All patients admitted to the acute care, non-ICU inpatient floors in the hospital between July 1, 2012, and December 31, 2019, were included in the study. Our institution is a 364-bed academic, quaternary-care, freestanding children’s hospital. Information from patients admitted to 1 of the hospital’s ICUs was excluded from the analysis while the patient was in the ICU, but information of any inpatient floor stays before or after an ICU stay was included to reflect real-world conditions.

The study was reviewed and approved by our institutional review board (2019–2465, approved December 10, 2018). The reporting of this study followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement and checklist.¹²  Data were analyzed using MATLAB 2019b (MathWorks, Natick, MA) and R version 4.0 (R Project for Statistical Computing, Vienna, Austria). A P value of <.05 was considered statistically significant.

To perform the model development and later implementation, we considered the input from a multidisciplinary team a key requirement for success. Therefore, we convened a team of clinicians, data scientists, quality improvement experts, and clinical informaticians. The team met regularly throughout the planning and execution of the project.

Data from 2012 to 2018 were analyzed retrospectively to evaluate the baseline model and develop the final prediction model. Data were extracted from the institutional data warehouse using structured queries and underwent quality checks for conformity, completeness, and plausibility.

Patients with sepsis on the inpatient units were identified using the IPSO Collaborative intention-to-treat criteria.¹³  Specifically, the IPSO intention-to-treat criteria were defined as treatment consistent with sepsis: intravenous antibiotics, ≥2 intravenous fluid boluses, or ≥1 bolus and a vasoactive agent within 6 hours and a blood culture within 72 hours.¹³  The IPSO criteria also include screening tools, huddles, and order sets as optional, site-specific criteria, but because we had no preexisting screening or huddle workflow in the inpatient setting, and there was very inconsistent use of related order sets, these were not included as criteria.

We first evaluated the performance of the inpatient sepsis prediction model developed at Nationwide Children’s Hospital, which is now available in the Epic Foundation system.¹⁴  This model includes routinely collected data in the EHR such as vital signs, nursing assessments, laboratory test results, presence of indwelling lines, and specific high-risk conditions ascertained by Systematized Nomenclature of Medicine concepts.¹⁵  Retrospective application of the Nationwide model for our institution yielded a sensitivity of 83% and a number needed to alert (NNA, calculated as 1/positive predictive value [PPV]) of 23 at the encounter level and did not meet our goals for implementation for an NNA of 10 or less for bedside evaluations.¹⁶ 

We chose to leverage the variables identified by the Nationwide model and train a new model. To better capture the idiosyncrasies of our patient population, which includes high degree of medical complexity, the high-risk conditions included in the original Nationwide model (ie, history of solid organ transplant, cancer, sickle cell, encephalopathy, cerebral palsy, severe intellectual disability, immunodeficiency, and medical technology dependency) were disaggregated and evaluated as separate candidate predictors instead of as a composite single variable. The final list of predictor variables evaluated included: lethargy, prolonged capillary refill, fever or hypothermia, systolic hypotension, tachycardia, tachypnea, weak pulse exam, cool/cyanotic/pale skin exam, leukocytosis or leukopenia, bandemia, elevated alanine aminotransferase, history of solid organ transplant, immunodeficiency, immunosuppression, chemotherapy, medical technology dependency, presence of a central line or urinary drain, and admission within the last 12 hours. Lethargy was based on the Pediatric Early Warning Score Behavior assessment charted by the bedside nurse. Of note, none of the variables used in the model included any of the ones used as criteria for the IPSO intention-to-treat sepsis definition to avoid biasing the model (ie, simply detecting when the medical team has started treating sepsis). Following the approach of the Nationwide model, continuous variables were dichotomized or categorized to facilitate implementation in the EHR. The high-risk conditions (history of solid organ transplant, cancer, sickle cell, encephalopathy, cerebral palsy, severe intellectual disability, immunodeficiency, and medical technology dependency) were based on diagnosis groupers. Immunosuppression and chemotherapy were based on prescribed medications.¹⁴ 

Inpatient encounters from 2012 to 2018 were used as a derivation set to train the model. The derivation set was structured into a right alignment framework for sepsis cases with controls matched by length of stay.¹⁷  In a right-aligned framework, samples are usually considered until the event of interest (for the positive cases) or the end of admission (for the negative cases).¹⁷  In our approach, variables were collected for the 24-hour time window preceding the time 0 for sepsis cases (on the basis of the IPSO criteria) and the corresponding time window for controls with similar length of stay matched in a 1:30 ratio. For example, a patient with sepsis at 35 hours after admission would have the values of the predictor variables extracted for the time window between 9 hours and 34 hours of admission. At the same time, 30 control patients with a length of stay of 35 or more hours would have the values for the same predictor variables extracted for the same time windows. This approach was taken to train the model by more closely mimicking real-world conditions and avoid training an overly optimistic model by using the predischarge time window for control patients, as is often performed in right alignment frameworks.¹⁷  Missing values were assumed to be normal (eg, if the alanine aminotransferase level was not measured, it was imputed as a normal value) or the condition absent (eg, if there was no diagnosis code for solid organ transplantation, the patient was assumed to not have that high-risk condition). In each 24-hour prediction time window, the worst value for each variable (eg, lowest systolic blood pressure, highest heart rate, etc) was used in the model.

Using the right aligned framework, a logistic regression model was trained and assessed using 40-fold cross-validation. Variables with a nonsignificant association with the outcome in the full model were excluded from the final model. The final model coefficients were transformed into an integer score for implementation by multiplying the value by 5 and rounding to the nearest integer.

A 2-tier threshold approach was determined using a precision-recall curve of the model: an “Alert” tier with a goal of having a PPV ≥10% (and an NNA ≤10 or less) to prompt clinical team bedside evaluation, and an “Aware” tier with high sensitivity of ≥80% to increase situational awareness.

Prospective validation was considered a critical step in the process. The reproducibility of the model outputs may be affected by real-world conditions such as erroneous or delayed data entry in the EHR by bedside clinicians and timeliness of variables, such as the diagnosis groupers used for classifying high-risk conditions.¹⁸  Therefore, the model was implemented in silent mode (ie, with no results provided to bedside clinicians) in the EHR, and scores from the year 2019 were used as prospective validation before live implementation.

In the prospective validation set, the performance was assessed on the basis of the maximum sepsis score at 2 levels: the encounter level, where the time periods were the time from admission to sepsis for cases or until discharge for controls; and at the 24-hour level, where time periods were all the 24-hour periods from admission until sepsis for cases or until discharge for controls.

A total of 55 980 inpatient encounters and 793 (1.4%) episodes of sepsis on the inpatient units between 2012 and 2019 were used for derivation (2012–2018) and prospective validation (2019) of the model. Patient and encounter characteristics including age, sex, race/ethnicity, preferred language, payer, and specific high-risk conditions for sepsis were similar between the training and validation periods (Table 1). The cumulative incidence of sepsis in the derivation set was slightly higher than in the prospective validation set (1.5% [n = 725] vs 1% [n = 68]).

The final model consisted of 13 routinely collected clinical variables including vital signs, nursing assessments, underlying high-risk conditions, and laboratory test results. Table 2 presents the variables in the final mode, the criteria for each variable, the model coefficients, and the integer score used for implementation. A precision-recall curve was used to set actionable thresholds (Fig 1). In the derivation set, the model had an area under the curve of 0.82, the Aware threshold had 87% sensitivity for sepsis, and the Alert threshold had a PPV of 10% (NNA of 10).

In the prospective validation set, the model had an area under the curve of 0.96 (95% confidence interval 0.95–0.97) at the encounter level and 0.91 (95% confidence interval 0.87–0.94) within 24 hours of sepsis onset (Table 3). The Aware threshold had 100% sensitivity for sepsis at the encounter and 24-hour levels, and the Alert threshold had a PPV of 14% (NNA of 7) at the encounter level and a PPV of 5% (NNA of 20) at the 24-hour level (Table 3).

We developed and prospectively validated a prediction model of sepsis in the non-ICU inpatient setting. Our model was designed to have 2 tiers of actionable thresholds, a high sensitivity threshold to increase situational awareness and a low NNA threshold to reduce the risk of alert fatigue.

The low incidence of sepsis in the non-ICU inpatient population, counterbalanced by the possible severe consequences of a missed or delayed diagnosis of severe sepsis or septic shock, make the development and implementation of sepsis prediction models challenging in the pediatric inpatient setting. EHR-based sepsis prediction tools have been shown to have poor predictive performance when used without validation before implementation.^19–21  Our multidisciplinary team employed a novel approach to create an automated, data-driven, EHR-based, 2-tiered sepsis prediction model to implement as a data-driven CDS tool with tailored responses and specific workflows, as presented in the companion paper.¹¹ 

By creating a 2-tiered sepsis prediction model, we aimed to maintain high situational awareness for sepsis while minimizing alert fatigue and stewarding resources. Alert fatigue is a known barrier to sepsis screening and is a pervasive threat to CDS implementation in health care.^19,22  Additionally, although bedside sepsis huddles have been impactful, they do come at a cost of the time clinical teams spend performing and documenting huddles.^23,24  Thresholding the Alert tier by NNA allowed for consideration of false alerts, alert fatigue, and resources required for a sepsis huddle when designing our model.¹⁶  The Aware tier with high sensitivity was designed to enable situational awareness and prompt discussions about sepsis risk at the individual patient, clinical team, and unit level. Situational awareness or “watcher” initiatives have demonstrated success at decreasing emergency ICU transfers and reducing serious safety events using proactive risk identification, mitigation, and escalation systems.^25–27  Applying the fundamental theorem of biomedical informatics, where the data-driven decision support tools serve to augment rather than replace clinician judgement, was instrumental when planning how to derive and implement our sepsis prediction model.²⁸ 

Our study has several strengths. First, we used a large, single-center derivation set to develop our model. Although this will likely reduce the generalizability of the model to other institutions (eg, our population has a high degree of medical complexity compared with other pediatric settings), we believe it maximizes its internal validity and effectiveness for our local implementation, which was the intended goal. Additionally, we believe that prospectively validating the model in real time represents a particular strength of our approach. The reproducibility of prediction models can be affected by the real-world conditions.¹⁸  For example, erroneous or noisy data can significantly affect model performance, particularly if the data used for derivation underwent significant data cleaning procedures. Furthermore, the timing when variables are available in real time (eg, diagnosis codes) may also affect model performance.

Our study also has several limitations. First, we used the IPSO intention-to-treat criteria for sepsis for model development, which may overestimate true sepsis incidence (ie, by capturing “overtreated” patients). Although the IPSO intention-to-treat sepsis criteria have known limitations, we believed it was important for us to stay aligned with the IPSO Collaborative approach given that the evaluation of our implementation efforts would be centered around the same sepsis criteria and the associated metrics used by the IPSO Collaborative. One risk of taking such approach is that the model is likely skewed toward greater sensitivity and lower specificity of true sepsis, but given that the goal of the prediction model is to increase shared situational awareness and leverage the team-based huddles and clinician evaluation of alerted patients to ascertain the presence of sepsis, this is a tradeoff that we deemed acceptable. It is also possible that the intention-to-treat sepsis criteria could miss cases of true sepsis on the acute care floors in which none of the treatment-based criteria were met while the patient was still on the acute care floor. Again, we believed that it was important for us to stay aligned with the IPSO Collaborative approach. Second, when evaluating our model and defining sensitivity and NNA thresholds, we did this by using the right-aligned, encounter-level data framework. Although this approach presents a true ratio of cases and controls that meet the model thresholds, it does not account for all the 24-hour periods when sepsis criteria are not being met in both cases and controls, as reflected in Table 3. From a practical standpoint, the right-aligned, encounter-level framework answers the question, “How many patients who subsequently met sepsis criteria had a score above the alert or aware thresholds compared to patients without sepsis?” whereas the 24-hour framework answers the question, “How many patients had a score above the Alert or Aware thresholds in 24-hour periods when no sepsis occurred subsequently?” The first question is primarily focused on the fundamentals of the model performance; that is, whether the predictors included do in fact predict sepsis or not, whereas the second question is primarily focused on the implementation practicalities, such as how many alerts would be expected with a 24-hour lockout period before a sepsis case is subsequently detected. Although the 24-hour period was chosen for pragmatic reasons given our a priori plan to have a 24-hour lockout period after an alert was triggered during clinical implementation, it is unclear what the risk–benefit ratio may be for alerts that are triggered before 24 hours in patients who eventually develop sepsis. Perhaps earlier awareness and vigilance may reduce the incidence of sepsis overall, or could potentially lead to precocious or unnecessary interventions. Unfortunately, such questions can only start to be answered after clinical implementation. Finally, we chose to leverage the same approach as the Nationwide model, which is now used in the Epic Foundation, to facilitate model development and implementation in the EHR. For example, we evaluated the same list of candidate predictors, we also used the worse values for each variable in the 24-hour prediction time window preceding the prediction time point, and categorized continuous variables into the same bins. However, it is possible that other sets of variables or features of the same variables (eg, change in heart rate over time) may be more predictive of sepsis.

We derived and prospectively validated a 2-tiered prediction model of pediatric sepsis designed to have a high sensitivity Aware threshold to enable situational awareness and a low NNA Alert threshold to minimize the rate of false alerts. Our model was embedded in our EHR and underwent live implementation in the form of a CDS tool and associated clinical workflows, which are presented in the companion article.¹¹