Impact of Automated Sepsis Metric Evaluation on Provider Performance
Impact of Automated Sepsis Metric Evaluation on Provider Performance
Sepsis is a life-threatening condition caused by the body's response to infection and is a leading cause of death worldwide. Hospitals use a complex quality measure called SEP-1 to track whether patients with severe sepsis or septic shock receive recommended care, such as timely antibiotics, fluids, and laboratory testing. However, evaluating SEP-1 is difficult. It requires manual review of medical records, is time-consuming and expensive, and typically provides feedback to clinicians months after care is delivered. This delay limits the ability to improve care in real time.
This study tested whether artificial intelligence (AI), specifically a type of system called a large language model (LLM), could improve the quality of sepsis care by providing faster and more detailed feedback to physicians.
The study was conducted at two emergency departments within a large academic health system. Sixty-six attending physicians were randomly assigned to one of two groups. In the intervention group, the AI system reviewed each patient's medical record at the time of hospital discharge and determined whether SEP-1 care standards were met. Physicians then received near real-time, individualized feedback about their performance, including specific areas for improvement. In the control group, physicians received standard feedback based on a small sample of cases reviewed months later using traditional methods.
Sepsis is a leading cause of morbidity and mortality worldwide and remains a major focus of hospital quality improvement efforts. In the United States, the Centers for Medicare & Medicaid Services (CMS) Severe Sepsis and Septic Shock Management Bundle (SEP-1) is a publicly reported quality measure that evaluates adherence to evidence-based processes of care, including timely antibiotic administration, fluid resuscitation, and laboratory testing. Despite its importance, SEP-1 is widely recognized as a complex measure, consisting of dozens of individual elements that must be completed within specific timeframes. Assessment of compliance typically relies on manual chart abstraction, which is resource-intensive, subject to variability, and performed on a small subset of cases with delays of several months. These limitations reduce the ability of SEP-1 reporting to drive timely improvements in clinical care.
Advances in artificial intelligence (AI), particularly large language models (LLMs), offer an opportunity to automate the extraction and interpretation of clinical information from unstructured medical records. Prior work has demonstrated that LLMs can achieve high agreement with expert reviewers when applied to complex clinical abstraction tasks. Building on this foundation, this study evaluates whether integrating AI-enabled chart abstraction into a real-world clinical workflow can improve performance on a complex quality measure by providing near real-time feedback to clinicians.
This study was conducted as a prospective, cluster randomized quality improvement initiative across two academic emergency departments within a single health system. Attending emergency physicians were randomized at the provider level to either an intervention group receiving AI-generated feedback or a control group receiving standard quality reporting feedback. Randomization at the physician level was selected to minimize contamination while preserving real-world clinical workflows.
All adult patients presenting to the emergency department who met CMS criteria for severe sepsis or septic shock were eligible for inclusion. Cases were identified using clinical encounter diagnoses and evaluated according to SEP-1 specifications. Each case was classified as meeting or failing the measure based on completion of required elements within defined time windows.
In the intervention arm, an LLM-based system automatically reviewed each eligible patient encounter at the time of emergency department discharge. The system analyzed structured and unstructured clinical data to determine SEP-1 compliance and identify specific elements of care that were incomplete or not documented. Physicians received individualized feedback shortly after patient care was completed. Feedback included a summary of the case, the determination of compliance, and targeted recommendations for improvement when deficiencies were identified. In selected cases, additional follow-up communication was provided to reinforce learning and clarify opportunities for improvement.
In the control arm, physicians received feedback through standard institutional processes. This consisted of manual chart abstraction of a limited number of cases selected for CMS reporting, with results communicated weeks to months after the clinical encounter. This approach reflects the current standard of care in most health systems.
The primary objective of the study was to evaluate whether near real-time, AI-enabled feedback improves SEP-1 compliance compared to standard delayed feedback. Secondary objectives included assessing the agreement between AI-generated determinations and expert human reviewers, as well as evaluating selected patient-centered outcomes, including intensive care unit (ICU) admission and 30-day mortality.
Because randomization occurred at the physician level and outcomes were measured at the patient encounter level, analyses accounted for clustering of patients within physicians. Mixed-effects modeling was used to estimate the effect of the intervention while accounting for this hierarchical structure.
The intervention is grounded in established principles of audit-and-feedback, a widely used strategy for improving clinician performance. Prior research suggests that audit-and-feedback interventions are more effective when feedback is timely, individualized, and actionable. The AI-enabled system in this study was designed to address known limitations of traditional feedback approaches by evaluating all eligible cases rather than a small sample, reducing delays between care delivery and feedback, and providing specific guidance tailored to individual clinician performance.
Importantly, this study focuses on improving adherence to a process-based quality measure rather than directly modifying clinical decision-making at the point of care. As such, the intervention is intended to support clinician learning and quality improvement without introducing real-time clinical decision support alerts that may contribute to alert fatigue.
This study also evaluates the feasibility of integrating AI tools into routine clinical operations at scale. By automating chart review and feedback generation, the intervention has the potential to reduce the administrative burden associated with quality reporting and to enable continuous performance monitoring across all eligible patients.
While improved compliance with SEP-1 may have implications for hospital quality metrics and reimbursement, the relationship between SEP-1 adherence and patient-centered outcomes remains an area of ongoing investigation. This study is not powered to detect differences in mortality or other clinical outcomes, but it provides an important step in understanding how AI can be leveraged to improve quality measurement and feedback processes.
The findings of this study may have broader implications beyond sepsis care. The ability of AI systems to extract clinical context from electronic health records and deliver timely, targeted feedback could be applied to a wide range of quality and safety measures. This approach may support the development of learning health systems in which data generated during routine care are continuously used to inform and improve clinical practice.
In summary, this study evaluates a novel application of AI to enhance quality measurement and feedback in sepsis care. By addressing key limitations of existing reporting systems, this approach aims to improve clinician performance on a complex quality measure and to establish a scalable model for AI-enabled quality improvement in healthcare.
Inclusion Criteria:
Exclusion Criteria: