AI-Assisted Chest-CT Reporting for Enhanced Speed and Quality (The DOUBLE-ACE Study)

A Multicenter Comparative Study Evaluating the Impact of an AI-Assisted Chest CT Reporting System on Real-world Radiologist Performance: The DOUBLE-ACE Study

The goal of this observational study is to learn if an AI assistant tool can help doctors who read chest CT scans (called radiologists) write their reports faster and just as well or better. Chest CT scans are common pictures taken of the inside of the chest to help with diagnosis. The main questions the study aims to answer are: (1) Does using the AI tool save radiologists time when writing their reports? (2) Are the final reports written with the AI tool's help as good as or better than reports written without it? To answer these questions, researchers will compare two time periods at several hospitals. They will look at how long it took to write reports and how good the reports were, both from a time before the AI tool was available and from a time after it was in regular use. In this study, radiologists will use the AI tool as part of their normal daily work. The tool is built into the computer system they already use to look at scans. Researchers will then measure the time and quality of the reports produced during their regular shifts.

Here we provide a summary of the study's methodological framework, including a description of the AI system under evaluation, key quality control measures, and the data analysis plan. Comprehensive details regarding the full protocol, including eligibility criteria and outcome measures, can be found in the other modules of the study protocol.

1. Background on the AI System: The study evaluates a clinically deployed AI-assisted reporting system, built upon an advanced multimodal foundation model trained on a large-scale chest CT dataset. Its performance, reliability, and generalizability have been established through rigorous validation on extensive internal and external datasets. Prior reader studies have demonstrated its clinical utility by significantly reducing reporting time through automated draft generation while maintaining or improving report quality, supporting its integration into real-world workflow for this evaluation.
2. Quality Control: To ensure objective assessment, report quality will be scored by a panel of at least two independent, blinded thoracic radiologists using a standardized rubric, with inter-rater reliability calculated. A study-specific data dictionary and Standard Operating Procedures (SOPs) for data handling and analysis will be implemented to ensure reproducibility and auditability.
3. Data Analysis Plan: A comparative statistical analysis between pre- and post-implementation groups is planned. Appropriate statistical tests (e.g., Mann-Whitney U test, mixed-effects models) will be applied based on data distribution and variable type. A sample size calculation will be conducted to ensure the study is adequately powered to detect a clinically meaningful difference in the primary efficiency endpoint. The primary analysis will be a paired comparison of outcomes (e.g., report time, quality) between the two phases for the participants who complete both. To address potential attrition (e.g., radiologist turnover during the study year) and the influence of radiologist experience, the analysis plan includes: (1) Accounting for and reporting any participant dropout between phases. (2) Conducting pre-specified subgroup or stratified analyses based on radiologist seniority (e.g., junior vs. senior) to examine its effect on the outcomes.
4. Confounding Factor Control: To minimize potential bias, the study may identify collective variables (radiologists' sex, years of relevant professional experience, etc.) that may be considered potential confounding factors according to external experts' judgements. Certain patient-related information, such as diagnosis (infection, malignancy, cardiovascular disease, etc.) and clinical scenario (e.g., inpatient, outpatient, emergency), may also be collected retrospectively, where necessary, to evaluate model performance within specific diagnostic subgroups. The study will adopt multiple possible approaches for confounding factor adjustment or analysis, which may include stratified analyses and other related statistical methods.
5. Potential Adjustments in Study Protocol or Analysis Methods: As the study may involve multiple centers, in case of ethical or administrative restrictions at certain time at specific sites, AI assistance may be temporarily suspended to approximate the scenario without AI assistance at those sites. In such cases, the corresponding results may be reported as the with-AI and without-AI phases, rather than labeling them as baseline and AI-available phases. Furthermore, radiologists who decline to provide demographic or occupational information (e.g., years of professional experience or sex)-variables that may serve as potential confounders-will be excluded from adjusted and stratified analyses that require such covariates. These approaches may need to be incorporated into confounding factor or stratified analyses, and we may update related conditions accordingly when necessary.

The study participants include both the radiologists whose performance is evaluated and the chest CT scans they interpret. Eligibility criteria are defined for both.

1. Inclusion Criteria

1.1 For Radiologists

1. Board-certified radiologists specializing in or routinely performing thoracic imaging.
2. Employed at one of the participating study centers for the entire duration of both the without-AI and with-AI study periods.
3. Interpreted a minimum of eligible chest CT scans (e.g., > 50 scans) during both the without-AI and with-AI data collection periods.

1.2 For Chest CT Scans

1. Non-contrast chest CT examinations performed for any clinical indication.
2. Scans completed and finalized during the defined with-AI or without-AI study periods.
3. Patient age 18 years or older at the time of the scan.

2. Exclusion Criteria

2.1 For Radiologists:

1. Radiologists who joined, left, or were on extended leave (e.g., >4 weeks) from the participating center between the with-AI and without-AI study periods.
2. Radiologists who interpreted fewer than the minimum required number of eligible scans in either study period.
3. Radiologists who voluntarily decline to have their de-identified performance data included in the study analysis.
4. Radiologists who decline to provide demographic or occupational information (e.g., years of professional experience or sex)-variables that may serve as potential confounders-will be excluded from adjusted and stratified analyses that require such covariates.

2.2 For Chest CT Scans

1. CT scans of pediatric patients (age < 18 years).
2. Contrast-enhanced chest CT studies.
3. Studies performed for specific procedural guidance (e.g., biopsy, ablation).
4. Studies deemed technically inadequate for primary interpretation by radiologist (e.g., severe motion artifact, incomplete study).
5. Studies for which the AI system fails to generate a valid preliminary report draft. This includes possible system failures, algorithm errors, or cases where the generated draft is deemed technically unusable (e.g., empty, garbled, or based on critically flawed image analysis).
6. The lack of relevant information (diagnosis, clinical scenario, etc.). Chest CT data will be excluded from corresponding analyses if the required information, which is necessary for confounding control, subgroup analyses, or other pre-specified analyses, is unavailable. Such scenarios include data that cannot be retrospectively retrieved, incompletely recorded, or restricted due to ethical or institutional requirements.