Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
The escalation of care for patients in a hospitalized setting between nurse practitioner managed services, teaching services, step-down units, and intensive care units is critical for appropriate care for any patient. Often such "triggers" for escalation are initiated based on the nursing evaluation of the patient, followed by physician history and physical exam, then augmented based on laboratory values. These "triggers" can enhance the care of patients without increasing the workload of responder teams. One of the goals in hospital medicine is the earlier identification of patients that require an escalation of care. The study team developed a model through a retrospective analysis of the historical data from the Mount Sinai Data Warehouse (MSDW), which can provide machine learning based triggers for escalation of care (Approved by: IRB-18-00581). This model is called "Medical Early Warning Score ++" (MEWS ++). This IRB seeks to prospectively validate the developed model through a pragmatic clinical trial of using these alerts to trigger an evaluation for appropriateness of escalation of care on two general inpatients wards, one medical and one surgical. These alerts will not change the standard of care. They will simply suggest to the care team that the patient should be further evaluated without specifying a subsequent specific course of action. In other words, these alerts in themselves does not designate any change to the care provider's clinical standard of care. The study team estimates that this study would require the evaluation of ~ 18380 bed movements and approximately 30 months to complete, based on the rate of escalation of care and rate of bed movements in the selected units.
Objectives:
Mount Sinai Hospital has developed a Rapid Response Team (RRT) system designed to give general floor care providers additional support for patients who may be requiring a higher level of care. This system enables both nurses and physicians to notify the RRT and have a critical care team evaluate the patients. During the period of 03/01/2018 to 09/17/2018, Mount Sinai Hospital floor units on 10W and 10E units made 357 rapid response team (RRT) calls with only 58 leading to an actual increase in the level of care (true positive rate ~ 16%). Similarly, the Electronic Health Record (EHR) generated 839 sepsis Best Practice Alerts (BPAs) yet only five led to escalations in care (true positive rate ~ 0.5%). The results above would imply that over 168 evaluations need to be made to identify a single case where the patient required an escalation in care. The goal of ReSCUE-ME is to evaluate prospective model performance and identify the best spot which the study team can incorporate MEWS++ into RRT and Primary providers workflow. The primary endpoint is rate of escalation of care on 10W and 10E during the study period.
Background:
In a prior study, the group has demonstrated that a machine learning model (MEWS++) significantly outperformed a standard, manually calculated MEWS score on a large retrospective cohort of hospitalized patients. To develop this model, the study team used a data set (Approved by the Program for Protection of Human Subjects Institutional Review Board (IRB) IRB-18-00581) of 96,645 patients with 157,984 hospital encounters and 244,343 bed movements. The study team found that MEWS++ was superior to the standard MEWS model with a sensitivity of 81.6% vs. 44.6%, specificity of 75.5% vs. 64.5%, and area under the receiver operating curve of 0.85 vs. 0.71.
Encouraged by this prior result, the study team is seeking to evaluate the model in a prospective study.
A silent pilot of the ReSCUE-ME alerts has been running on 10E and 10W since Feb 2019. The study team has continuously monitoring the alert performance via a real-time web-based dashboard. The results are summarized below:
Not provided
Not provided
Not provided
Not provided
| Label | Type | Description | Intervention Names |
|---|---|---|---|
| MEWS++ Monitoring | Active Comparator | This consists of all the patients that will be receiving MEWS++ escalation monitoring and provider alerting. |
|
| Standard of Care Monitoring | Placebo Comparator | Patients in the control arm will have a score calculated but no alert will be sent. |
|
| Name | Type | Description | Arm Group Labels | Other Names |
|---|---|---|---|---|
| MEWS++ Monitoring | Other | Patient's electronic medical record data will undergo processing by a machine learning algorithm (MEWS++). |
|
| Measure | Description | Time Frame |
|---|---|---|
| Overall Rate of Escalation | Rate of escalation of care from floor to Stepdown, Telemetry, ICU, per 1,000 patient bed days. | 10 months |
| Measure | Description | Time Frame |
|---|---|---|
| Number of Participants Requiring Blood Pressure Support | Number of participants requiring blood pressure support agents such as initiation of vasopressor medication or administration of fluid bolus. | 10 months |
| Number of Participants Requiring Respiratory Support |
Not provided
Inclusion Criteria:
Exclusion Criteria:
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Name | Affiliation | Role |
|---|---|---|
| Matthew A Levin, MD | Icahn School of Medicine at Mount Sinai | Study Director |
| Facility | Status | City | State | ZIP | Country | Contacts |
|---|---|---|---|---|---|---|
| Mount Sinai Hospital | New York | New York | 10029 | United States |
| PubMed Identifier | Type | Citation | Retractions |
|---|---|---|---|
| 38380992 | Derived | Levin MA, Kia A, Timsina P, Cheng FY, Nguyen KA, Kohli-Seth R, Lin HM, Ouyang Y, Freeman R, Reich DL. Real-Time Machine Learning Alerts to Prevent Escalation of Care: A Nonrandomized Clustered Pragmatic Clinical Trial. Crit Care Med. 2024 Jul 1;52(7):1007-1020. doi: 10.1097/CCM.0000000000006243. Epub 2024 Feb 21. |
Not provided
Not provided
Deidentified raw data may be shared with other researchers upon submission of a written request along with a plan for how the data are intended to be used.
Not provided
Not provided
Not provided
Not provided
All patients admitted to the study units were enrolled. There were no opt-outs.
Not provided
Not provided
| ID | Title | Description |
|---|---|---|
| FG000 | MEWS++ Monitoring | This consists of all the patients that will be receiving MEWS++ escalation monitoring and provider alerting. MEWS++ Monitoring: Patient's electronic medical record data will undergo processing by a machine learning algorithm (MEWS++). Predictor Score: A score predicting the likelihood that the patient will experience a deterioration in their clinical condition within six hours will be generated. If the prediction score exceeds a predetermined threshold, an alert will be sent to the provider. The alerting protocol is tiered, with both a low and high threshold. If the score is above the low threshold, nursing will be notified. If the score is above the high threshold, RRT will be notified. |
| FG001 | Standard of Care Monitoring | Patients in the control arm will have a score calculated but no alert will be sent. Predictor Score: A score predicting the likelihood that the patient will experience a deterioration in their clinical condition within six hours will be generated. If the prediction score exceeds a predetermined threshold, an alert will be sent to the provider. The alerting protocol is tiered, with both a low and high threshold. If the score is above the low threshold, nursing will be notified. If the score is above the high threshold, RRT will be notified. |
| Title | Milestones | Reasons Not Completed | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall Study |
|
Not provided
Not provided
| ID | Title | Description |
|---|---|---|
| BG000 | MEWS++ Monitoring | This consists of all the patients that will be receiving MEWS++ escalation monitoring and provider alerting. MEWS++ Monitoring: Patient's electronic medical record data will undergo processing by a machine learning algorithm (MEWS++). Predictor Score: A score predicting the likelihood that the patient will experience a deterioration in their clinical condition within six hours will be generated. If the prediction score exceeds a predetermined threshold, an alert will be sent to the provider. The alerting protocol is tiered, with both a low and high threshold. If the score is above the low threshold, nursing will be notified. If the score is above the high threshold, RRT will be notified. |
| Units | Counts |
|---|---|
| Participants |
|
| Title | Description | Population Description | Parameter Type | Dispersion Type | Unit of Measure | Calculate Percentage | Denominator Units Selected | Denominators | Classes |
|---|---|---|---|---|---|---|---|---|---|
| Age, Continuous | Mean |
| Type | Title | Description | Population Description | Reporting Status | Anticipated Posting Date | Parameter Type | Dispersion Type | Unit of Measure | Calculate Percentage | Time Frame | Units Analyzed | Denominator Units Selected | Arm/Group Information | Denominators | Classes | Analyses | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Primary | Overall Rate of Escalation | Rate of escalation of care from floor to Stepdown, Telemetry, ICU, per 1,000 patient bed days. | Posted | Number | escalations per 1,000 patient bed days | 10 months |
|
All-cause mortality data were collected up to 30 days post-admission from hospital, up to 6 weeks post-admission.
No adverse event data were collected other than all cause mortality.
Not provided
| ID | Title | Description | Deaths (Affected) | Deaths (At Risk) | Serious Events (Affected) | Serious Events (At Risk) | Other Events (Affected) | Other Events (At Risk) |
|---|---|---|---|---|---|---|---|---|
| EG000 | MEWS++ Monitoring | This consists of all the patients that will be receiving MEWS++ escalation monitoring and provider alerting. MEWS++ Monitoring: Patient's electronic medical record data will undergo processing by a machine learning algorithm (MEWS++). Predictor Score: A score predicting the likelihood that the patient will experience a deterioration in their clinical condition within six hours will be generated. If the prediction score exceeds a predetermined threshold, an alert will be sent to the provider. The alerting protocol is tiered, with both a low and high threshold. If the score is above the low threshold, nursing will be notified. If the score is above the high threshold, RRT will be notified. |
Not provided
Not provided
Not provided
| Title | Organization | Phone | Extension | |
|---|---|---|---|---|
| Dr. Matthew Levin | Icahn School of Medicine at Mount Sinai | 212-241-8382 | matthew.levin@mssm.edu |
Not provided
| Type | Includes Protocol | Includes SAP | Includes ICF | Document Label | Document Date | Document Uploaded Date | Document File Name |
|---|---|---|---|---|---|---|---|
| Prot_SAP | Yes | Yes | No | Study Protocol and Statistical Analysis Plan | Jul 28, 2022 | Mar 13, 2024 | Prot_SAP_000.pdf |
Not provided
| ID | Term |
|---|---|
| D000075902 | Clinical Deterioration |
| ID | Term |
|---|---|
| D018450 | Disease Progression |
| D020969 | Disease Attributes |
| D010335 | Pathologic Processes |
| D013568 | Pathological Conditions, Signs and Symptoms |
Not provided
Not provided
For each patient, real-time data from clinical and administrative systems will be used by ReSCUE-ME to produce a MEWS++ score predicting the likelihood that the patient will require escalation of care within the next 6 hours. Upon the patient being admitted to the unit, the patient will be evaluated based on any update in the EHR. If the prediction score exceeds a "high" threshold, the RRT team will be notified directly. If the score is between a "low" threshold and the high threshold , the nursing team will be notified and increased nursing monitoring will be initiated. If the patient has met criteria for increased nursing monitoring, a refractory 8-hour refractory window will be applied during which no nursing alerts will be sent. However if the score exceeds the high threshold, the RRT team will be notified. Throughout the trial, the performance of the alerts will be monitored via web-based dashboards. If the performance is poor, the "high" and "low" thresholds will be adjusted.
Not provided
Not provided
No masking is completed as the information/waiver of consent sheet for the two arms needed to be individualized.
Not provided
| Predictor Score | Other | A score predicting the likelihood that the patient will experience a deterioration in their clinical condition within six hours will be generated. If the prediction score exceeds a predetermined threshold, an alert will be sent to the provider. The alerting protocol is tiered, with both a low and high threshold. If the score is above the low threshold, nursing will be notified. If the score is above the high threshold, RRT will be notified. |
|
Number of participants requiring respiratory support intervention such as initiation of nasal cannula to high flow or frequency of intubation. |
| 10 months |
| Number of Participants Who Experienced a Cardiac Arrest Episode | The number of patients who had a cardiac arrest. | 10 months |
| Mortality Rate | Number of Mortalities - Combined In-hospital and 30-day mortality. Mortalities only counted once. 30-day mortality includes those patients who died in-hospital within 30 days. | Duration of hospital stay, until discharge, regardless of stay length for patients who died in hospital, or 30 days after admission, starting from date of admission, up to 6 weeks. |
| Notification Frequency - Number of Alerts Sent Per Day to Providers | The number is calculated as average number of alerts sent per day over all the days in the study period. | 10 months |
| Number of Calls | The average number of calls to RRT made per patient, regardless of alert. Not evaluated. | 10 months |
| Sensitivity and Specificity of the RRT Alert | The performance of the alert will be evaluated by calculating the sensitivity, specificity, positive predictive value, negative predictive value, precision, recall, and F1-score. This will be done both for the overall escalation rate and if possible for individual escalations (ICU, step-down, telemetry) and death. | 10 months |
| BG001 | Standard of Care Monitoring | Patients in the control arm will have a score calculated but no alert will be sent. Predictor Score: A score predicting the likelihood that the patient will experience a deterioration in their clinical condition within six hours will be generated. If the prediction score exceeds a predetermined threshold, an alert will be sent to the provider. The alerting protocol is tiered, with both a low and high threshold. If the score is above the low threshold, nursing will be notified. If the score is above the high threshold, RRT will be notified. |
| BG002 | Total | Total of all reporting groups |
| years |
|
| Sex: Female, Male | Count of Participants | Participants |
|
| Ethnicity (NIH/OMB) | Count of Participants | Participants |
|
| Race/Ethnicity, Customized | Count of Participants | Participants |
|
| BMI | Mean | Standard Deviation | kg per meters squared |
|
| Elixhauser Score | A measure of the overall severity of comorbidities, based on ICD diagnosis codes. Range is -19 to +89, with a score of > +15 indicating severe comorbid illness. | Mean | Standard Deviation | units on a scale |
|
| First Deterioration Score on Admission | Predicted likelihood of escalation to intensive care or death in hospital. This score is the output of the MEWS++ random forest model. Data from the EHR, laboratory information system and ECG system were used. Features included laboratory test results, vital signs, automated ECG interpretations, and structured clinical nursing documentation (e.g., level of consciousness). Full model description published in PMID 32012659. Range 0-1. Cutoff for a positive prediction 0.59. At this threshold the sensitivity was 0.4, specificity 0.78, PPV 0.13, and NPV 0.94 for ICU escalation or death. | Mean | Standard Deviation | proportion probability |
|
A score predicting the likelihood that the patient will experience a deterioration in their clinical condition within six hours will be generated.Patients in the control arm will have a score calculated but no alert will be sent.
|
|
|
| Secondary | Number of Participants Requiring Blood Pressure Support | Number of participants requiring blood pressure support agents such as initiation of vasopressor medication or administration of fluid bolus. | Posted | Count of Participants | Participants | 10 months |
|
|
|
|
| Secondary | Number of Participants Requiring Respiratory Support | Number of participants requiring respiratory support intervention such as initiation of nasal cannula to high flow or frequency of intubation. | No statistical comparison performed due to low rate of events. | Posted | Count of Participants | Participants | 10 months |
|
|
|
| Secondary | Number of Participants Who Experienced a Cardiac Arrest Episode | The number of patients who had a cardiac arrest. | No statistical comparison performed due to no events. | Posted | Count of Participants | Participants | 10 months |
|
|
|
| Secondary | Mortality Rate | Number of Mortalities - Combined In-hospital and 30-day mortality. Mortalities only counted once. 30-day mortality includes those patients who died in-hospital within 30 days. | Posted | Count of Participants | Participants | Duration of hospital stay, until discharge, regardless of stay length for patients who died in hospital, or 30 days after admission, starting from date of admission, up to 6 weeks. |
|
|
|
|
| Secondary | Notification Frequency - Number of Alerts Sent Per Day to Providers | The number is calculated as average number of alerts sent per day over all the days in the study period. | No statistical analysis performed | Posted | Mean | Standard Deviation | alerts per day | 10 months |
|
|
|
| Secondary | Number of Calls | The average number of calls to RRT made per patient, regardless of alert. Not evaluated. | Data were not collected for this outcome measure. | Posted | 10 months |
|
|
| Secondary | Sensitivity and Specificity of the RRT Alert | The performance of the alert will be evaluated by calculating the sensitivity, specificity, positive predictive value, negative predictive value, precision, recall, and F1-score. This will be done both for the overall escalation rate and if possible for individual escalations (ICU, step-down, telemetry) and death. | Patients in the intervention arm. | Posted | Number | proportion | 10 months |
|
|
|
| Post-Hoc | Time to ICU Escalation | Time in hours between alert and transfer to an ICU | Posted | Median | Inter-Quartile Range | hours | From time of alert until transfer to an ICU, assessed up to discharge from the hospital or death |
|
|
|
| Post-Hoc | Likelihood of Earlier Hospital Discharge | Hazard ratio for faster earlier discharge for patients who got an alert | Patients who survived to hospital discharge | Posted | Count of Participants | Participants | Duration of hospital stay, starting from admission, up to the day of discharge, regardless of the length of hospital stay, up to 1 year. |
|
|
|
|
| 104 |
| 1,488 |
| 0 |
| 0 |
| 0 |
| 0 |
| EG001 | Standard of Care Monitoring | Patients in the control arm will have a score calculated but no alert will be sent. Predictor Score: A score predicting the likelihood that the patient will experience a deterioration in their clinical condition within six hours will be generated. If the prediction score exceeds a predetermined threshold, an alert will be sent to the provider. The alerting protocol is tiered, with both a low and high threshold. If the score is above the low threshold, nursing will be notified. If the score is above the high threshold, RRT will be notified. | 117 | 1,252 | 0 | 0 | 0 | 0 |
Not provided
Not provided
Not provided
| 30-d mortality |
|
| RRT Alerts |
|
| Title | Measurements |
|---|---|
|
| Negative Predictive Value |
|