Preliminary Evaluation of a Large Language Model-Based Tool for Complex Surgical Decision Support in Lung Cancer

Preliminary Evaluation of a Large Language Model-Based Tool for Complex Surgical Decision Support in Lung Cancer

This study is an exploratory effect-size estimation study, with the following specific objectives: ① to estimate the point estimate and 95% confidence interval of the Win Ratio for the experimental group (GAPS-Agent) versus the control group (large language model) in blinded pairwise preference judgments by thoracic surgery expert adjudicators, to serve as a sample size planning parameter for subsequent multicenter confirmatory clinical trials; ② to preliminarily evaluate the value of GAPS-Agent within clinical workflows.The hypothesis of this study is as follows: compared with a general-purpose large language model without medical enhancement (control group), a structured agentic workflow optimized on the basis of the GAPS evaluation framework (GAPS-Agent, experimental group) can help junior resident physicians generate clinical decision plans for complex lung cancer cases that are more strongly preferred by senior thoracic surgery expert adjudicators.

Inclusion Criteria:

1. Resident Physician Subjects:

   1. Holds a valid and legally effective Physician Practice License of the People's Republic of China;
   2. Currently holds the rank of resident physician in a thoracic surgery department at a tertiary Class A (3A) hospital;
   3. Agrees to complete all assessment tasks of the main study phase in accordance with the study protocol;
   4. Can guarantee the time and effort required to complete all assessment tasks of the main study.

2. Study Cases:

   1. The case was discussed at the Thoracic Oncology Multidisciplinary Team (MDT) conference of Peking University People's Hospital between January 2025 and May 2026;
   2. The current version of the NCCN guidelines does not provide an explicit recommendation covering the management of the case;
   3. Does not overlap with the GAPS evaluation set;
   4. The case is presented in pure text in a structured format, with all direct and indirect identifiers removed and complete de-identification performed prior to inclusion;
   5. From the pool of eligible cases, 12 cases will be randomly drawn using Python (numpy.random, with a fixed and archived seed) to serve as the main study cases. The cases will cover 6 themes (chest mass of undetermined diagnosis, early-stage lung cancer, locally advanced lung cancer, oligometastatic/oligoprogressive disease, special intraoperative situations, and tumor recurrence), with 1 - 4 cases per theme.

3. Adjudication Expert Panel:

   1. Holds a valid and legally effective Physician Practice License of the People's Republic of China;
   2. Currently holds the rank of attending physician or above in a thoracic surgery department at a tertiary Class A hospital;
   3. Chairs or regularly participates in lung cancer multidisciplinary team (MDT) work in their department.

Exclusion Criteria:

1. Resident Physician Subjects:

   1. Has previously participated in the construction of the GAPS evaluation set or the development of GAPS-Agent;
   2. Unable to complete the tasks of the study phase.

2. Study Cases:

   1. Key case information is missing, such as text-form data on pathology (including IHC/NGS), imaging, laboratory tests, prior medical history, comorbidities, or PS score;
   2. Decision-making for the case is strictly dependent on non-text information.

3. Adjudication Expert Panel:

   1. Participated in the construction of the GAPS evaluation set, the content validity verification, or the development of GAPS-Agent for this study;
   2. Has a direct conflict of interest with any specific product among the two-arm tools of this study.