Effect of a Localized ICU-Specific AI Teaching Agent on Institutional Workflow Mastery and Clinical Competency in Rotating ICU Residents: A Single-Center, Parallel-Group, Randomized Controlled Superiority Trial
Effect of a Localized ICU-Specific AI Teaching Agent on Institutional Workflow Mastery and Clinical Competency in Rotating ICU Residents: A Single-Center, Parallel-Group, Randomized Controlled Superiority Trial
This trial is an ongoing single-center, pragmatic, parallel-group randomized controlled superiority trial currently in participant recruiting phase, conducted within intensive care unit teaching wards at Peking Union Medical College Hospital, Beijing, China. The scheduled trial implementation period spans March 2026 to June 2026, aiming to evaluate whether an institution-specific, protocol-bound retrieval-augmented AI educational agent (named ICU-Tutor) can reduce residents' extraneous cognitive load and improve standardized ICU protocol task performance compared with free access to unrestricted commercial general-purpose large language model AI tools during early ICU clinical rotation.
The trial plans to screen a total of 44 first-time ICU rotating resident candidates, with pre-defined exclusion standards to eliminate unqualified individuals; approximately 44 eligible residents will undergo 1:1 stratified randomization and be split into two research arms: 22 participants assigned to the ICU-Tutor intervention group and 22 assigned to the unrestricted general AI control group.
All enrolled subjects will complete standardized 14-day follow-up assessments as pre-specified in the trial protocol. Both study cohorts receive unified 15-minute standardized training covering standardized safe AI clinical application rules prior to formal intervention initiation. ICU-Tutor is strictly built on a curated knowledge base including 247 ICU institutional protocols validated by senior attending intensivists, with all AI outputs traceable back to original local protocol documents and constrained within verified institutional guidance content only. The control arm allows participants to select and utilize any mainstream general large-model AI tools per personal preference without content or access limitations, consistent with real-world daily resident clinical practice.
Two co-primary endpoints are uniformly scheduled to be measured on the 7th day after randomization, including total completion duration of standardized ICU protocol task battery and Paas 9-point validated cognitive load scale score reflecting participants' subjective mental workload during task execution. Three confirmatory secondary endpoints are pre-defined for centralized assessment: composite task performance score on Day7, written institutional protocol knowledge retention score tested on Day14, and 0-100-point visual analog scale (VAS) evaluating resident satisfaction toward allocated AI support on Day7. Individual sub-station scores of three split practical ICU skill modules are set as exploratory secondary endpoints for post-hoc descriptive analysis only.
The statistical analysis framework is pre-specified to follow intention-to-treat principle entirely. Analysis of covariance (ANCOVA) is selected as core analytical method for all continuous outcomes, with Day3 baseline assessment result and participants' academic training background set as pre-planned covariates. Bonferroni multiple-testing correction is applied for dual co-primary endpoints, while Benjamini-Hochberg false discovery rate (FDR) correction is pre-specified to control type I error across three confirmatory secondary outcomes. Effect sizes will be quantified via Cohen's d after raw data collection and database lock.
The trial has obtained formal ethical approval from the Institutional Review Board of Peking Union Medical College Hospital (Approval ID: I-26ZM0024). Every enrolled resident provides written informed consent before random assignment.
Research Background Early-stage intensive care unit (ICU) clinical rotation imposes substantial cognitive challenges for newly incoming resident physicians. Novice residents must adapt to unfamiliar ward workflows, manage critically ill patients with unstable vital status under severe time constraints, and master a large volume of hospital-specific institutional clinical protocols covering routine and emergency ICU management. Rooted in Cognitive Load Theory, unnecessary extraneous cognitive workload originating from repeated information searching and cross-verification severely occupies limited working memory capacity and hinders clinical learning efficiency during high-intensity residency training.
General-purpose large language model (LLM) AI tools have gained widespread popularity as supplementary learning resources among medical trainees worldwide. Nevertheless, conventional off-the-shelf LLMs lack embedded access to hospital-specific, site-validated ICU clinical protocols, which leads to generalized, decontextualized recommendations inconsistent with local institutional practice requirements. Consequently, residents are forced to spend extra working memory to validate AI-generated suggestions against internal hospital guidelines, generating avoidable cognitive burden that impedes on-the-job protocol learning and bedside task execution.
Retrieval-augmented generation (RAG) framework enables customized institutional AI agents bounded exclusively within locally approved ICU protocols, delivering source-cited, site-compliant clinical guidance without requiring post-hoc manual verification by trainees. Existing medical education AI research predominantly evaluates model performance on generalized medical knowledge examinations rather than real-world on-site protocol application within authentic ICU working environments. This prospective randomized controlled trial is designed to fill this research gap by comparing the educational benefits of a hospital-customized protocol-locked AI agent (ICU-Tutor) versus unrestricted free access to commercial general LLMs among first-time ICU rotating residents during a standardized 14-day observation window. The core research hypothesis specifies that ICU-Tutor will lower resident cognitive load and improve protocol-based task performance relative to open-access general AI.
Predefined Study Objectives Primary Objectives To prospectively evaluate whether ICU-Tutor reduces standardized task completion time and self-reported Paas-scale cognitive load on study Day 7 compared with unrestricted general-purpose AI access in first-time ICU rotating residents. These two metrics serve as co-primary trial endpoints for formal statistical comparison.
Confirmatory Secondary Objectives
Three pre-specified secondary endpoints will undergo formal statistical testing with pre-defined multiple-testing correction rules:
Compare composite standardized practical task performance scores between the two study arms at post-randomization Day 7; Quantify inter-group differences of institutional ICU protocol knowledge retention via closed-book written assessment administered on Day 14; Measure participant satisfaction differences toward assigned AI support tools using a 0-100 visual analogue scale (VAS) collected on Day 7.
Exploratory Objective Separate individual score analysis across three independent practical assessment stations including airway management, hemodynamic assessment and ventilator troubleshooting will be implemented as exploratory analyses only. No formal statistical correction will be applied for these sub-station outcomes, whose aggregated total score constitutes the composite Day7 task performance primary secondary endpoint. All exploratory findings are intended for hypothesis generation rather than confirmatory conclusion.
Trial Design and Study Setting This is a prospective, single-center, pragmatic, parallel-group superiority randomized controlled trial currently in active participant recruitment, scheduled to run from March 2026 through June 2026 at ICU clinical teaching units of Peking Union Medical College Hospital, Beijing, China, a large tertiary academic medical center. The pragmatic trial design preserves residents' routine daily ICU clinical responsibilities throughout the entire research period without modification to regular clinical rotation arrangements; all assigned AI interventions are integrated into authentic daily clinical workflow consistent with standard real-world residency training practice. The trial protocol complies with core methodological and reporting standards from CONSORT 2010 and the CONSORT extension statement for pragmatic randomized trials. Institutional Review Board (IRB) approval (approval code I-26ZM0024) has been fully obtained prior to recruitment launch.
Participant Eligibility, Screening and Enrollment Plan Inclusion Criteria First-time ICU rotation participants with zero prior formal ICU clinical rotation experience; Scheduled to complete a minimum of four consecutive weeks of ICU training after trial enrollment; Capable of completing all scheduled multi-timepoint assessments (Day3 baseline, Day7 core endpoint tests, Day14 retention evaluation); Possess basic digital device operation competency to access assigned AI platforms during daily clinical work; Voluntarily provide written informed consent for full trial participation. Exclusion Criteria Accumulated prior formal ICU clinical experience exceeding seven calendar days; Regular daily clinical utilization of various commercial AI learning tools within three months preceding enrollment screening; Physical or cognitive impairment prohibiting full completion of all required trial assessments across pre-defined follow-up timepoints.
Screening & Randomization Target Sample Size The research team plans to consecutively screen a total of 44 incoming rotating residents who initiate ICU rotation within the March-June 2026 recruitment window. Per pre-trial protocol estimation, all screened candidates are anticipated to satisfy pre-set eligibility criteria without exclusion, leading to full randomization of 44 qualified participants at a fixed 1:1 allocation ratio: 22 subjects assigned to the ICU-Tutor intervention group and another 22 participants allocated to the unrestricted general-AI control group. Recruitment activities are completed during centralized departmental resident orientation sessions hosted by the hospital's Division of Critical Care Medicine.
Randomization Strategy and Blinding Arrangement Eligible enrolled participants receive computer-generated variable-block randomization with stratified allocation to guarantee balanced baseline distribution across study groups. Pre-specified stratification covariates include postgraduate training year and baseline AI proficiency, with AI experience graded via a validated 1-5 Likert scale split into low experience (score 1-2) and high experience (score 3-5) subgroups. The fixed 1:1 grouping ratio yields planned n=22 per trial arm as defined in the trial statistical plan.
Complete participant blinding is not feasible due to distinct functional differences between the customized ICU-Tutor system and heterogeneous commercial general AI software products, and all residents are aware of their assigned AI tool category after randomization. To minimize assessment bias, all attending physicians responsible for grading standardized practical station assessments remain fully blinded to individual participants' group assignment throughout endpoint evaluation. Independent research coordinators exclusively manage randomization codes and participant grouping information, entirely separated from clinical assessors responsible for outcome scoring.
Detailed Description of Study Interventions All participants from both intervention and control cohorts attend a uniform 15-minute pre-intervention orientation covering standardized safe AI application specifications. The unified training explicitly emphasizes that all AI-derived content serves only as auxiliary clinical reference and cannot replace attending physician clinical judgment or mandatory institutional ICU policies.
Intervention Arm (ICU-Tutor, Planned n=22) ICU-Tutor is built on a strict RAG architecture anchored to a curated institutional knowledge repository containing 247 ICU-specific clinical protocols comprehensively peer-reviewed and validated by senior attending intensivists from the host hospital's critical care department. The covered protocol spectrum includes mechanical ventilation management, hemodynamic support, sedation and analgesia regulation, continuous renal replacement therapy (CRRT), antimicrobial stewardship codes, in-hospital emergency response algorithms, ICU enteral/parenteral nutrition guidance, delirium screening and intervention, venous thromboembolism prophylaxis, and end-of-life institutional care protocols.
A conservative safety constraint is embedded within ICU-Tutor's core algorithm: all system-generated responses are strictly limited to information retrieved exclusively from pre-approved internal protocol documents, with each output attached with traceable source links referencing original institutional guideline files. The hospital's ICU clinical governance committee assumes ongoing responsibility for periodic repository updates to maintain protocol currency throughout trial implementation.
Control Arm (Unrestricted General-Purpose AI, Planned n=22) Control-group participants receive full unrestricted permission to select any commercially available mainstream general LLMs per individual personal preference, matching routine off-protocol resident AI usage in real clinical practice. No constraints are imposed on tool type, daily access frequency or usage duration for control subjects. Consistent with pragmatic trial design goals, centralized backend usage log collection is not pre-scheduled for control participants given the heterogeneous assortment of self-selected third-party AI platforms.
Pre-specified Outcome Metrics and Assessment Timetable All assessments follow fixed pre-defined post-randomization timelines: baseline testing on study Day3, core primary plus confirmatory secondary endpoint measurements on Day7, and delayed knowledge retention written examination on Day14.
Co-Primary Endpoints (Uniform Day7 Assessment) Total task completion time (in minutes): cumulative time spent finishing a composite standardized ICU protocol task battery consisting of three independent practical assessment stations; Paas 9-point cognitive load scale score: validated single-item subjective rating ranging from 1 (extremely low mental workload) to 9 (extremely high mental workload), measuring overall perceived cognitive burden generated during standardized protocol task completion.
Confirmatory Secondary Endpoints Day7 composite practical task performance score (0-100 total points): aggregated total score combined from three separate practical station evaluations graded by blinded attending physicians using unified standardized checklists; Day14 institutional protocol knowledge retention score (percentage scoring): closed-book written examination covering core content of the hospital's verified ICU protocols; Day7 AI satisfaction VAS (0-100): continuous visual analogue scale measuring participants' overall satisfaction with their allocated AI auxiliary resource.
Exploratory Endpoints Independent discrete scoring results for three individual practical sub-stations: airway management (max 40 points), hemodynamic assessment (max 30 points), ventilator troubleshooting (max 30 points). These discrete sub-station results are solely for exploratory post-hoc analysis and are not defined as formal confirmatory trial endpoints. Supplementary self-reported learning efficiency questionnaires will be collected descriptively without pre-planned inter-group statistical comparison.
Full Trial Implementation Timeline Pre-recruitment preparation (prior to March 2026): Finalize IRB document filing confirmation, ICU-Tutor system debugging, standardized assessment test bank construction and evaluator training for blinded attending raters; Active screening & enrollment phase (March 2026 - target completion before mid-June 2026): Rolling resident eligibility screening, informed consent acquisition and formal randomization for all 44 planned participants; Baseline evaluation (Day3 post-randomization): All randomized residents complete baseline protocol knowledge scoring and pre-trial AI experience questionnaire; Formal intervention period (Randomization Day through Day14): Continuous daily authorized access to assigned AI tools following group allocation during routine ICU clinical work; Core endpoint centralized assessment (Day7): Unified collection of co-primary outcomes plus confirmatory secondary endpoint measurements; Delayed follow-up assessment (Day14): Administer closed-book knowledge retention written examination for all enrolled subjects; Post-study administrative phase (after June 2026): Complete raw data aggregation, database lock and pre-specified statistical analysis per analytic plan once full participant follow-up finishes.
Predefined Statistical Analysis Plan All statistical analyses will strictly adhere to the intention-to-treat (ITT) principle; every randomized participant remains in their originally allocated study arm regardless of subsequent AI usage compliance after enrollment. After full data collection, continuous outcome indicators will be summarized with standard descriptive statistics, while categorical baseline variables will be reported via frequency and proportion for inter-group baseline balance verification.
Analysis of Covariance (ANCOVA) is pre-specified as the primary statistical model for all continuous confirmatory endpoints, incorporating study grouping as fixed independent factor alongside two pre-defined covariates: Day3 baseline knowledge score and resident academic training background (undergraduate, master's or doctoral clinical training track). Bonferroni correction is pre-applied to the two co-primary endpoints to adjust familywise type I error rate; Benjamini-Hochberg false discovery rate (FDR) correction will be implemented across three confirmatory secondary outcomes to mitigate multiple testing bias. Cohen's d standardized mean difference is pre-selected as uniform effect-size metric for all between-group comparisons after database lock and data finalization.
Per original trial protocol design, full participant follow-up is anticipated for all enrolled 44 subjects, hence no pre-planned multiple imputation strategy for missing outcome data is defined. All exploratory sub-station score comparisons will use unadjusted statistical testing without any multiple-testing correction, with findings interpreted only for future hypothesis development.
Ethical Governance and Informed Consent Framework The entire trial has secured formal ethical approval from Peking Union Medical College Hospital Institutional Review Board (IRB approval ID: I-26ZM0024). Prior to any screening and randomization activities, all potential participants receive detailed written and verbal briefings outlining trial purpose, intervention characteristics, assessment schedule and associated potential risks and benefits. Written informed consent documentation is mandatorily obtained before formal study enrollment.
Every enrolled resident reserves unconditional right to voluntarily withdraw trial participation at any timepoint without punitive consequences affecting their routine ICU rotation evaluation or clinical training progress. All ICU-Tutor generated outputs are prominently labeled as AI-derived auxiliary guidance to avoid excessive clinical reliance by residents, fully complying with pre-defined clinical safety specifications. No identifiable patient protected health information is collected or processed during ICU-Tutor operation or full trial execution.
Pre-identified Anticipated Trial Limitations
Several inherent structural limitations are prospectively acknowledged within the finalized trial protocol prior to participant recruitment initiation:
Single-center trial setting restricts external generalizability of final trial results to ICUs with divergent institutional protocols or distinct resident training systems at other medical institutions; Limited 14-day follow-up window cannot capture long-term sustained learning improvement or persistent clinical competency shifts after completion of early ICU rotation; No patient-level clinical outcome indicators are embedded within the trial design, prohibiting evaluation of indirect AI-associated influences on real-world patient care metrics; Non-blinded participants create potential risk of novelty or expectation bias impacting subjective scoring for cognitive load and satisfaction VAS endpoints; Heterogeneous self-selected general AI tools within control arm without unified usage log collection restricts secondary exploratory analysis linking AI usage frequency to measured outcomes; additionally, long-term sustainable ICU-Tutor operation requires continuous clinical governance to update institutional protocol repositories and monitor rare potential AI output inaccuracies.
Funding and Conflict of Interest Disclosure This clinical investigation is financially supported via internal institutional research funding from Peking Union Medical College Hospital. The funding body has no authority or involvement in trial protocol design, participant recruitment, raw data acquisition, subsequent statistical analysis or future manuscript drafting activities. All study investigators pre-declare no relevant financial or commercial competing interests linked to the ICU-Tutor AI intervention being assessed in this trial. Following full trial completion and institutional administrative approval for data sharing, de-identified aggregate trial dataset will be available for supplementary research review upon formal request.
Inclusion Criteria:
Exclusion Criteria:
zhouyuankai@pumch.cn+86-10-69152300