Evaluation of Contextflow DETECT Lung CT Nodule Detection Software in Chest CT Scans

Evaluation of Computer-Aided Lung Nodule Detection Software in Chest CT Scans With an Assessment of Its Impact on Readers Decision-Making Process

contextflow DETECT Lung CT is a Artificial Intelligence (AI)-based computed-aided detection (CADe) system, intended to support radiologists in the detection of lung nodules in chest computed tomography (CT) scans. System is intended to be used as a second-reader, therefore results provided by the software are meant to complement the radiologist's findings and decisions.

Proposed study will be multi-reader, multi case (MRMC) retrospective reader study. The goal of the study is to evaluate the influence of CADe on the effectiveness of lung nodule detection. During the study, 10 radiologists will analyze 350 chest CT scans of adult patients, with and without the assistance of CADe. The study will be conducted remotely. CT scans will be uploaded to a web-based image submission and annotation platform, in which every participant of the study will be provided with individual account and assigned task list.

The primary objective of the study determine if the diagnostic accuracy of radiologists with CADe assistance is superior to the diagnostic accuracy of radiologists without CADe assistance in localizing the pulmonary nodules with enhanced area under the free-response operating characteristic curve (AUC of FROC).

The study will target approximately 350 asymptomatic adult patients, whose CT scans were acquired during routine CT examination. The patient population will include patients with and without lung nodules.

1. Background and rationale

   According to the American Cancer Society, lung cancer was the second commonly diagnosed cancer with an estimated 2.2 million new cases in 2020. Currently, the most effective way of reducing cancer mortality is early diagnosis of lung cancer. However, in routine clinical practice, many early-stage lung cancer were delayed in diagnosis, due to the fact that most patients are often asymptomatic. That's the main reason why fast and the least invasive lung disease screening is so important for the early detection of pathological changes in asymptomatic patients from high-risk cohorts. Computed tomography (CT) is the most recent dominated imagining tool used to capture lung images, and therefore also for nodule detection and characterization. Consequently, computer-aided detection and software systems based on artificial intelligence have become one of the most prominent and valuable tools for detection of pulmonary nodules.

   The aim of the proposed reader study is to evaluate one of such tools, contextflow DETECT Lung CT, and to assess to what extent supporting radiologists with a computer-aided detection system improves the detection of nodules and supports radiologists in the decision-making process.

   contextflow DETECT Lung CT is an AI-based web application for 3D medical imaging data. It's intended to be used for the detection and visualization of pulmonary nodules in chest CT scans of clinically asymptomatic adult patients.

   The system is intended to alert radiologists to specific locations in which nodules can be detected that may otherwise be overlooked. The system provides measurements of the nodules, such as average diameter, long-axis diameter, short-axis diameter as well as the total volume.

   contextflow DETECT Lung CT is intended to be used as a second reader, in addition to original reading performed by radiologists. Results provided by contextflow DETECT Lung CT are intended to complement the information that radiologists take into account during their decision-making process.

2. Study design

   This will be a retrospective, multi-reader multi-case (MRMC) study. The performance of radiologists alone will be compared with the performance of radiologists supported by the detection software (as a second reader). The study will include 10 readers, and a final dataset of 350 scans collected from at least 3 radiology centers and revised by truthers. During the study three following reads are planned: Read 1 (Truthing), Read 2 (Baseline, without CADe assistance), Read 3 (Intervention, with CADe assistance).

   The study will consist of 350 CT images collected from asymptomatic adult patients. At least 50% of the cases shall be US data, other cases will be collected from European radiology centers. At least 65% of the cases will include at least one lung nodule. The dataset will also include cases without nodules.

3. Selection of readers

   Readers qualification criteria:

   • US-board of Radiology or equivalent specialty certification
   • At least 5 years of chest CT interpretation experience
   • Fellowship-trained in thoracic/chest imaging,
   • Successful training on the use of study software.

4. Selection of patients and imaging data

   Patients, whose chest CT images were included into the dataset, must meet all the following criteria:

   • Adult patients (aged 22 or older) with the number of nodules ranged from 0 to 10;
   • For patients with at least one lung nodule: nodules detected and confirmed in the ground-truthing procedure;
   • Asymptomatic patients, who undergo a routine chest CT scan.

   Imaging data must meet all the following inclusion criteria to be in this study:

   • Images are CT images of the chest showing full left and right lung;
   • Images do not show any movement artifacts of the patient;
   • Images of the chest must be reconstructed in axial plane;
   • The number of nodules per image was 10 or fewer;
   • All nodules measuring between 4 - 30 mm in maximal long-axis diameter at CT scan.

5. Reader study procedures

   Ground truth process workflow: 3 US board-certified thoracic radiologists with at least 5 years experience after corresponding fellowship will be asked to read CT scans and identify actionable nodules in images from ground truthing dataset (370 CT scans randomly chosen from balanced candidate database) by marking them by manually drawing bounding boxes around them.

   Reading process workflow: For each CT scan, the reading process will take place in two steps: (1) reader's stand-alone reading, without contextflow DETECT Lung CT assistance and (2) second reader's reading with contextflow DETECT Lung CT as a second-reader. In each step, readers will analyze 350 CT scans dataset, with limited clinical information.

6. Blinding and randomization

   The identification of cases selected for the study, as well as individual analyzes between readers will be blinded to readers. Readers will not have access to the results from truthing procedure or to the information how many actionable nodules have been localized in individual CT images.

   Image submission and annotation platform will generate the random task list for each reader. The reader order will be different for all of the readers.

7. Study objectives and endpoints

   The primary and secondary objectives of the reader study is to demonstrate enhanced diagnostic accuracy of radiologists with CADe in comparison with the diagnostic accuracy of radiologists without CADe in

   Primary

   • Localizing the lesions with the enhanced area under the free-response receiver operating characteristic curve (AUC of FROC)

   Secondary

   • Detecting disease using (AUC of ROC)
   • Enhancing disease identification with AUC of LROC

   The following endpoints will also be evaluated:

   • The difference in sensitivities of lesion detection with and without CADe given specified average false positive numbers of lesions per image;
   • The difference in sensitivities of disease diagnostic with and without CADe given specified specificity;
   • Positive predictive value for disease diagnosis
   • Negative predictive value for disease diagnosis

8. Data management

   All data that is part of the study, both shared by our partners or downloaded from public sources, was anonymized following HIPAA and/or GDPR regulations and does not include any protected health information.

9. End of the study

The reader study will be considered completed when a sufficient number of images have been analyzed to achieve a satisfactory statistical test performance in the means of statistical power. To achieve that, at least 300 scans should be analyzed.

Inclusion Criteria:

• adult asymptomatic patients, who undergo a routine chest CT scan.

Exclusion Criteria:

• symptomatic patients.