Effect of Artificial Intelligence-Augmented Human Instruction on Surgical Simulation Performance

Effect of Artificial Intelligence-Augmented Human Instruction on Surgical Simulation Performance: A Randomized Controlled Trial

At the Neurosurgical Simulation and Artificial Intelligence Learning Centre, we seek to provide surgical trainees with innovative technologies that allow them to improve their surgical technical skills in risk-free environments, potentially improving patient operative outcomes. The Intelligent Continuous Expertise Monitoring System (ICEMS), a deep learning application that assesses and trains neurosurgical technical skill and provides continuous intraoperative feedback, is one such technology that may improve surgical education.

In this randomized controlled trial, medical students from four Quebec universities will be blinded and randomized to one of three groups (one control and two experimental). Group 1 (control) will be provided with verbal AI tutor feedback based on the ICEMS error detection. Group 2 will be tutored by a human instructor who will receive ICEMS error data and deliver verbal instruction using the same words as the ICEMS. Group 3 will be tutored by a human instructor who will be provided with ICEMS data and will then deliver personalized feedback.

The aim of this study is to determine how the method of delivery of verbal surgical error instruction influences trainee technical skill acquisition and transfer. Evaluating trainee responses to AI instructor verbal feedback as compared to feedback from human instructors will allow for further development, testing, and optimization of the ICEMS and other AI tutoring systems.

Background: Expert surgical technical skill is linked with improved patient outcomes; however, training novices to master these skills remains challenging. The Intelligent Continuous Expertise Monitoring System (ICEMS) is a deep learning application that was developed at the Neurosurgical Simulation and Artificial Intelligence Learning Centre to improve neurosurgical education. The ICEMS assesses and trains bimanual surgical performance by providing continuous feedback via verbal instructions in order to improve trainee performance and mitigate errors.

Rationale: A previous randomized controlled trial (RCT) performed at our centre demonstrated that intelligent tutoring is more effective than expert tutoring in a simulated neurosurgical procedure (NCT05168150). However, during this study, expert instructors were not provided with ICEMS error data. Conducting a new RCT in which expert instructors are provided with quantitative ICEMS error data will allow us to determine the most effective method for teaching surgical technical skills to trainees in virtual operative procedures.

This report follows the Consolidated Standards of Reporting Trials-Artificial Intelligence (CONSORT-AI) as well as the Machine Learning to Assess Surgical Expertise (MLASE) checklist.

Hypotheses:

1. AI-augmented personalized expert instruction will be more effective at improving technical skill acquisition and skill transfer in trainees compared to AI tutor instruction.
2. Instruction delivered by human instructors will result in lower levels of negative emotions and cognitive load compared with instruction delivered by the AI tutor.

Primary Objectives: To determine how the method of delivery of surgical error instruction influences:

1. Trainee technical skill acquisition and overall surgical performance (composite expertise scores across all practice tasks calculated by the ICEMS).
2. Trainee skill transfer to a more complex realistic scenario (composite expertise scores during realistic task calculated by the ICEMS).

Secondary Objective: To determine how the method of delivery of surgical error instruction influences trainee affective-cognitive responses, including emotions-self-reported via questionnaires administered before, during, and after learning using the Medical Emotions Scale (MES) on 7-point Likert scales-and cognitive load-self-reported via questionnaire administered after learning using the Cognitive Load Index (CLI) on 5-point Likert scales.

Setting: McGill University's Neurosurgical Simulation and Artificial Intelligence Learning Centre.

Participants: Students enrolled in their preparatory, first, or second year at one of four Quebec medical schools.

Design: A three-arm randomized controlled trial.

Intervention: Participants will undergo a training session of approximately 90 minutes on the NeuroVR (CAE Healthcare), a virtual reality (VR) surgical simulator that simulates a subpial brain tumor resection. In this study, participants will perform two different scenarios on the NeuroVR: a simple practice scenario and a complex realistic scenario. Participants will perform six repetitions of the practice scenario (5 minutes each) followed by the realistic scenario (13 minutes). The ICEMS will continuously assess performance throughout the trial. All participants will receive verbal feedback when the ICEMS detects an error in their performance; however, the method of delivery of this verbal feedback will differ between groups.

• Group 1 (control) will receive real-time verbal feedback directly from the ICEMS when an error is detected.
• Group 2 (experimental) will receive real-time verbal feedback from an expert instructor delivered in the same words as the ICEMS.
• Group 3 (experimental) will receive real-time, personalized verbal feedback from an expert instructor delivered in their own words.

Verbal feedback will be based on the following six metrics:

1. Tissue injury risk: When a trainee receives feedback on this metric, the healthy brain tissue has been damaged or is at risk of being damaged.
2. Bleeding risk: When a trainee receives feedback on this metric, there is bleeding that must be cauterized or a risk of bleeding.
3. Instrument tip separation distance: Refers to the distance between the tip of the ultrasonic aspirator and the tips of the bipolar forceps. When a trainee receives feedback on this metric, their instruments are too far apart.
4. High bipolar force: Refers to the amount of force applied to the tissue by the bipolar forceps. When a trainee receives feedback on this metric, they are applying too much force with the bipolar.
5. Low bipolar force: Refers to the amount of force applied to the tissue by the bipolar forceps. When a trainee receives feedback on this metric, they are not applying enough force with the bipolar.
6. High aspirator force: Refers to the amount of force applied to the tissue by the ultrasonic aspirator. When a trainee receives feedback on this metric, they are applying too much force with the aspirator.

These metrics will continuously be evaluated by the ICEMS. The ICEMS will only provide feedback on one metric at a time according to a predetermined hierarchy (in the order listed above). For example, if the ICEMS detects on error on both bleeding risk (2) and high aspirator force (6) at the same time, the system will only provide feedback on bleeding risk since this metric is above high aspirator force in the hierarchy.

One of neurosurgical residents who will instruct participants is completing an MSc at the Neurosurgical Simulation and Artificial Intelligence Learning Centre and the other two are conducting simulation research at our center. All three have had extensive experience performing subpial resection procedures using the NeuroVR simulator and our ex vivo calf brain model, as well as in the operating room. As such, these individuals are considered competent experts for the purpose of this simulation trial. Prior to the start of the trial, pilot studies were conducted. Each resident instructed three to five participants during these pilot studies. They were each evaluated by a senior consultant neurosurgeon with decades of clinical experience with subpial resections and subsequently identified as experts for their ability to instruct novice participants.

Study Procedure: Prior to the simulation session, the study coordinator will stratify participants according to their year in medical school and block randomize them to one of three intervention arms with an allocation ratio of 1:1:1 using random number sequences generated by Random.org. Upon arrival, participants will read and sign an informed consent form. They will then fill out a pre-trial questionnaire recording demographic information and self-reported baseline emotions using the MES. Trial instructions introducing the NeuroVR simulator, the instruments, and the practice subpial resection scenario will be provided via a written document followed by a video. Each practice task will last 5 minutes, followed by a 5-minute rest period. No post-hoc feedback will be provided during the rest periods. Participants will perform their first practice task without feedback to establish a baseline. Participants will then perform their second through fifth practice tasks while receiving their assigned educational intervention. This metric-specific action-oriented feedback will be provided according to ICEMS error detection. The sixth repetition of the practice task will serve as a summative assessment wherein participants will receive no feedback. Following the completion of the practice tasks, participants will complete a peri-trial questionnaire to assess their emotions using the MES. They will be provided with another information document introducing the realistic subpial brain tumor resection task. Participants will complete the 13-minute realistic task to assess their skill transfer to a more complex scenario. Finally, they will fill out a post-trial questionnaire assessing their emotions using the MES and their cognitive load using the CLI.

Significance: With surgical education approaches beginning to shift towards competency-based frameworks, the implementation of effective AI educational feedback into surgical training becomes crucial for optimizing surgical learning. The results of this RCT will allow for the evaluation and reengineering of the ICEMS and other AI tutoring systems, which may advance the development of not only standardized competency-based surgical education training curricula, but any AI tutor technology dependent on verbal instruction.

Inclusion Criteria:

• Medical students who are actively enrolled in their preparatory, first, second year of medical school at any Quebec institution who do not fit the exclusion criteria.

Exclusion Criteria:

• Prior use of the NeuroVR (CAE Healthcare) simulator.