Assessing Intensive Care Unit (ICU) Indications: Human vs. ChatGPT-4o Predictions

Evaluation of the Accuracy of Intensive Care Unit (ICU) Admission Indications in Emergency Department Patients: A Comparison Between Clinical Decisions and ChatGPT-4o Prediction

This retrospective study evaluates the accuracy of ICU admission indications by comparing clinical decisions with predictions from ChatGPT-4. Patient data, including demographics, vital signs, laboratory results, imaging findings, and clinical decisions, will be retrospectively collected and documented systematically using Case Report Forms. The model will be trained using ICU admission guidelines and tasked to predict ICU needs based on collected patient data. This study aims to systematically assess the alignment between AI-based predictions and clinical decisions for ICU admissions.

This study has a retrospective design. The medical data of patients admitted to the emergency department and consulted to the anesthesiology and reanimation clinic for ICU indications will be collected retrospectively. Demographic information, vital signs, laboratory results, imaging findings, and clinical decisions of the patients will be recorded. These data will be systematically collected for each patient using an individual Case Report Form.

Inclusion Criteria for the Study:

Patients aged 18 years and older who were consulted to the anesthesiology and reanimation clinic for ICU indications from the emergency department will be included in the study.

Exclusion Criteria for the Study:

Patients consulted to the anesthesiology and reanimation clinic for ICU indications from inpatient services.

Patients consulted to the anesthesiology and reanimation clinic from the emergency department for reasons other than ICU indications.

Patients consulted to the anesthesiology and reanimation clinic for ICU indications from the emergency department but with insufficient recorded data (patients with data loss).

Model Training and Prediction Analysis:

ChatGPT-4 will be trained according to the guidelines in "Yoğun Bakım Hasta Kabul Kriterleri (Rehberleri)" by Çiftçi B, Erdoğan C, and Demiraran Y (5). The collected patient data will be presented to the ChatGPT-4 model to obtain predictions regarding whether the patients require ICU admission. The predictions made by ChatGPT will be compared with clinical decisions, and accuracy rate, false positive rate, and false negative rate will be analyzed.

Statistical Analysis Methods to Be Used in the Study:

Accuracy Rate: The rate at which ChatGPT correctly predicts ICU indications will be calculated.

False Positive Rate: The rate at which ChatGPT predicts ICU need for patients who do not require ICU admission will be evaluated.

False Negative Rate: The rate at which ChatGPT predicts no ICU need for patients who require ICU admission will be analyzed.

Kappa Statistics: The agreement between ChatGPT predictions and clinical decisions will be measured.

ROC Curve and AUC: The performance of ChatGPT will be evaluated using the ROC curve and AUC.

The Case Report Form used for each patient ensures detailed and systematic data collection of clinical information, aiming to meaningfully compare the alignment of ChatGPT's predictions with clinical decisions.

Inclusion Criteria:

• Patients aged 18 years and older who are consulted to the anesthesiology and reanimation clinic for ICU indications from the emergency department will be included in the study.

Exclusion Criteria:

• Patients consulted to the anesthesiology and reanimation clinic for ICU indications from inpatient services.
• Patients consulted to the anesthesiology and reanimation clinic from the emergency department for reasons other than ICU indications.
• Patients consulted to the anesthesiology and reanimation clinic for ICU indications from the emergency department but with insufficient recorded data (patients with data loss)