Evaluation of V5 Lung AI Software With A Two-Arm Comparative Reader Study
Evaluation of V5 Lung AI Software With A Two-Arm Comparative Reader Study
STUDY DESIGN:
This is a two-arm retrospective, multi-reader, multi-case, (MRMC) randomized reader study.
OBJECTIVE:
Primary: The primary objective of this clinical study is to prove that radiologist's performance aided with V5med Lung AI is superior to the unaided for detecting qualified lung nodules.
Secondary: The secondary objective of this clinical study is to prove that the radiologist's reading time is not significantly increased when aided with V5med Lung AI.
Addition Objectives: To prove that the agreement (e.g., in kappa correlation coefficient) between experts and radiologist's Lung-RADS score aided with V5med Lung AI is superior to the unaided.
NUMBER OF SUBJECTS:
Retrospective CT studies from approximately 350 patients will be included in the study with approximately 170 true positive cases and 180 normal cases.
PRIMARY ENDPOINTS:
Scores given by the radiologists with and without V5med Lung AI will be recorded and compared to the true status of the study-cases. The frequency of the scores for each method (Aided, Unaided) will be tabulated and LROC curves constructed along with sensitivity, specificity, PPV, NPV and clinical actions. Additionally, machine nodule detection rate and false positives per patient on normal cases will be measured.
PATIENT POPULATION:
The study will target approximately two hundred (200) patients whose CT lung nodules were shown to be cancer and one hundred and eighty (150) patients who have no lung nodule greater than 4mm. The patient population will be consistent with the national lung cancer screening protocols.
Overview of the Study:
A reader study with sixteen participating radiologists will be conducted. A location-specific receiver operating characteristic (LROC) curve will be used to evaluate radiologists' detection performance in detecting qualified lung nodules on chest CT scans with and without the use of the V5med Lung AI system. The reading time for each case will also be recorded.
The 16 study radiologists will be split into two groups, X and Y. The 350 chest CT scans will be split into two subsets, A and B. Group X radiologists will interpret subset A without AI and subset B with V5med Lung AI. Group Y radiologist will interpret subset B without AI and subset A with AI. After a minimum of one month, each radiologist group will interpret the same images again but in the reverse order with regard to study modes (without AI and with AI). Please see Figures 1 and 2 for detailed arrangement of the radiologist interpretation sessions.
During the baseline reading session the radiologist will mark each location of qualified nodules and assign a score. If at least one suspicious nodule is observed, the radiologist will also indicate a Lung-RADS score (i.e., 2, 3, 4A, 4B, or 4X) at the end of case interpretation.
During the second reading (i.e., aided reading) session, the radiologist will be presented with a chest CT with CADe marks displayed on the "left window" and the same chest CT without CADe marks displayed on the "right window". If the radiologist confirms a nodule, the radiologist will mark the location on the chest CT displayed on the right window. This may or may not correspond to the locations of the CADe marks as displayed on the left window. The same as in the baseline study, the radiologist will assign a score (level of suspicion) to each suspicious location and provide a Lung-RADS score for the chest CT case, if at least one location on the CT is marked.
Based on the suspicious locations and their scores (level of suspicion) assessed by study radiologists, LROC curves can be constructed for both the baseline and the aided reads, and the significance of any difference will be calculated. The LOS score will also be used to calculate sensitivity, specificity, PPV and NPV.
Number and types of Chest CT cases:
Retrospective chest CT image series from approximately 350 cases will be included in the study. Approximately two hundred (200) cases will have at least one nodule size greater than 4mm. These cases will have radiology-prove report and/or biopsy-confirmed cancers. Also included as nodule images are those where a nodule was not detected at that time, but was detected and acted on based on a subsequent CT. These are the prior images where the nodule can be identified in the same location as on the "current" image confirmed by the radiologist expert panel (using a majority of three as the decision criterion). In addition, approximately one hundred and fifty (150) cases will not have a nodule size greater than 4mm.
The selected chest CT cases for the 2-arm comparative reader study, randomly selected from a larger pool of CT cases, will be enriched in the following way:
The Design of Two-Arm Comparative Study:
The First Arm Study:
Perform a baseline reader study without any special tool. The reader is asked to mark all potential nodules not less than 4mm. For each mark, the size of region of interest (ROI), reader score (level of suspicion), and time spent will be recorded for the statistical analyses.
The Second Arm Study:
Perform another reader study using the V5med Lung AI as the aided read. The reader is asked to mark all potential nodules not less than 4mm assisted by V5med Lung AI. Some or majority of ROIs' parameters will be pre-filled by V5med Lung AI. The reader will make final determination for each location. The marked location, LOS score, Lung-RADS score, and time spent will be recorded for the statistical analyses and compared to the first arm study.
The primary study hypothesis is that the detection of lung nodules with the use of V5med Lung AI and associated functions is superior to the detection of lung nodules without the use of V5med Lung AI, as measured by the area under the LROC curve.
Inclusion Criteria:
Exclusion Criteria: