AI-support in Emergency Medicine

Evaluation of AI Support for Differential Diagnosis in Emergency Medicine.

The purpose of this study is to investigate whether the use of generative artificial intelligence (AI) as a support tool can improve physicians' ability to establish accurate differential diagnoses in emergency care.

The main questions it aims to answer are:

• Is AI-assisted differential diagnosis more accurate than the differential diagnosis performed by physicians?
• Does physicians' differential diagnosis change - in terms of the number of correct diagnoses - when they have access to AI-assisted differential diagnosis?

The study is based on a prospective cohort of patients admitted through the emergency departments at Karolinska University Hospital in Huddinge and Solna. After completion of the hospital length of stay, excerpts from the medical record are collected according to a predefined variable list and used to create case descriptions corresponding to the information available at the initial assessment. Within this cohort, physicians will be randomized to evaluate patient cases either with or without AI support. The AI tool generates a structured list of likely differential diagnoses. Differential diagnoses proposed by physicians, AI alone, and physicians using AI support will be compared with the clinically established final diagnosis in the discharge summary. In a pilot material, the AI support has shown good agreement with final diagnoses and expert clinical assessment.

The results will be analyzed to assess AI's impact on physicians' diagnostic accuracy and decision-making in various scenarios and case presentations.

Given known challenges in diagnosing complex patients, the study is designed to account for variations in information quality and clinical complexity. Patient cases are presented in a standardized format, and it will be documented how much and what type of information is available in each case, such as symptoms, clinical findings, laboratory results, and disease progression. The analysis focuses on how this may affect the diagnostic suggestions. Difficult-to-diagnose cases are also included to clarify in which clinical situations the decision-support tool is useful, when it is not, and what informational requirements need to be met ahead of a future implementation study evaluating the impact of real-time AI support during the management of acutely ill patients.

There is a lack of large clinical studies that systematically evaluate how generative AI affects physicians' differential diagnostic performance in emergency medicine when used as a complement to clinical judgment. This project addresses this knowledge gap by examining the extent to which AI-generated differential diagnoses align with physicians' assessments and whether access to AI support can improve diagnostic precision.

The results will provide evidence on how AI-based decision support should be evaluated and potentially implemented within emergency care.

Methods:

Study Population Patients admitted to the hospital via the emergency department at Karolinska University Hospital in Huddinge and Solna. The plan is to collect data from 500 patients. The sample size was calculated using G*Power version 3.1.97, with a significance level (alpha) of 0.05, statistical power (1 - beta) of 0.8, and an expected effect size of 0.25, resulting in 210 patient cases per group. To compensate for potential loss to follow up and missing data, the planned included patients is set to 500.

Inclusion criteria: adult patients admitted to the hospital and discharged with a primary diagnosis in the discharge summary. Exclusion criteria: pediatric patients and cases lacking a discharge diagnosis. Emergency "red alert" patients will also be included. Patients will receive written and verbal information about the study and provide informed consent prior to participation. The patients may withdraw at any time. Patient inclusion will begin in the spring 2026. For validation, 30 patient cases from the study "ChatGPT and Generating a Differential Diagnosis Early in an Emergency Department Presentation" will be used.

Data Collection Patient data will be retrieved from the medical record, including the emergency department admission note, any ambulance report, diagnostic workup, and the discharge summary with final diagnosis. Collected data include medical history, vital signs, physical examination findings, laboratory results, imaging results, preliminary diagnosis, and-when applicable-differential diagnoses. After collection, all data will be pseudonymized.

AI Model and IT Security API calls are used to submit structured input, including the user's data (history, physical findings, test results, and point of care investigations). The input structure is based on iterative testing and simulations to ensure that the AI model performs as intended. In addition to patient data, specific prompts instruct the AI model to generate a structured list of differential diagnoses, ranked with the most likely diagnosis first. The code and prompts have been developed to optimize the model's ability to propose accurate differential diagnoses based on patient information. A customized version of a generative language model is used via Microsoft Azure OpenAI Service under the agreements of Karolinska Institutet and Karolinska University Hospital. All data processing occurs in Sweden in accordance with applicable laws and regulations.

Assessment Approximately 30 clinically active resident and specialist physicians in emergency medicine and internal medicine will review the patient cases. Physicians are randomized to assess cases either with or without access to the AI model's suggestions. Patient cases are presented to both physicians and the AI model in a standardized format developed through simulation and iterative refinement.

Data Processing and Analysis

The final diagnosis is defined as the primary diagnosis documented in the discharge summary. Descriptive statistics will be used for background characteristics of physicians and patients, as well as outcome distributions. Differences in diagnostic accuracy between groups will be analyzed using one way ANOVA. Logistic regression will be used to adjust for covariates and calculate odds ratios. Agreement between physicians and the AI model will be assessed using Cohen's kappa. Data on 30 day mortality will also be requested from the national Cause of Death Register.

Inclusion Criteria:

• Adult patients >18 years admitted through the emergency departments at Karolinska University Hospital in Huddinge and Solna.

Exclusion Criteria:

• Children <18 years
• Patients without a recorded final diagnosis