Retrospective Multicenter Multi-Reader, Multi-Case Diagnostic Accuracy Study of Carebot AI MMG Compared With Radiologists on 2D Full-Field Digital Mammography in Breast Cancer Screening
Retrospective Multicenter Multi-Reader, Multi-Case Diagnostic Accuracy Study of Carebot AI MMG Compared With Radiologists on 2D Full-Field Digital Mammography in Breast Cancer Screening
This study evaluates the diagnostic performance of Carebot AI MMG, an artificial intelligence (AI)-enabled medical device for evaluating mammograms. The software analyzes standard full-field digital mammography (FFDM) images and classifies each examination as having no suspicious finding ("Low Risk"), a probably benign mass ("Medium Risk"), or a suspicious malignant mass ("High Risk").
The study is retrospective and observational. It uses anonymized mammography examinations from four screening centers, without any additional imaging or contact with patients. Three experienced breast radiologists independently read the same set of cases, and their assessments are used as the human benchmark. A histopathology-based reference standard, supplemented by radiologist consensus and follow-up information for negative cases, is used to determine whether cancer is present.
The main goal is to compare the AI system with human radiologists in terms of sensitivity and specificity for detecting breast cancer, and to assess whether the AI can achieve non-inferior performance at two predefined operating points: one favoring higher sensitivity and negative predictive value (rule-out) and one favoring higher specificity and positive predictive value (rule-in).
Design and setting This is a retrospective, multicenter, multi-reader, multi-case (MRMC) diagnostic accuracy study of Carebot AI MMG, conducted on anonymized 2D full-field digital mammography (FFDM) examinations acquired as part of routine breast cancer screening. Mammograms were collected from four screening centers over a defined time period. No additional imaging was performed for the purpose of this study, and no subjects were contacted.
Data source and population The source dataset consists of 4,729 screening mammography examinations from women aged 32 to 88 years (mean approximately 57 years). Only 2D FFDM studies with a complete set of standard projections (LCC, RCC, LMLO, RMLO) were included. Examinations with incomplete series, unreadable or corrupted DICOM files, or missing/inconsistent key metadata were excluded, as were tomosynthesis (DBT) studies, men, and women under 18 years of age. To ensure sufficient precision of performance estimates, the dataset was enriched with additional biopsy-proven cancers. The final analytical subset comprises 222 examinations, including 48 malignant and 174 non-malignant studies, with representation across three mammography devices (Hologic Selenia Dimensions, Hologic Lorad Selenia, Fujifilm FDR-3000AWS).
Investigational device and comparator The investigational device is Carebot AI MMG (software version 2.9, deep-learning models v2.3), a stand-alone AI system that analyzes 2D FFDM exams and outputs a case-level classification into three categories, together with an internal risk score. Two predefined operating points are evaluated: a high-sensitivity (HSe) threshold, where both benign and malignant masses are treated as "positive" (rule-out setting), and a high-specificity (HSp) threshold, where only malignant masses are counted as "positive" (rule-in setting).
As a human comparator, three experienced radiologists (RAD 1-3) independently read the same anonymized studies using a dedicated DICOM viewer integrated with a labeling application. Radiologists were blinded to AI outputs, clinical information, and outcomes and recorded a case-level classification into the same three categories (Negative/Benign/Malignant). For primary analyses, their binary decisions are derived using the same HSe and HSp rules. In addition, a "random reader" benchmark is constructed for balanced accuracy by repeatedly sampling one radiologist's decision per case in a bootstrap framework (20,000 iterations).
Reference standard The reference standard is established at the study level. A case is labeled "Malignant" if there is histopathological confirmation of breast cancer from biopsy performed in temporal association with the index mammogram. A case is labeled "Non-malignant" if there is consensus between two local radiologists that the finding is negative or stably benign, typically corroborated by at least 2 years of imaging follow-up. Tumor staging (e.g., TNM) is not used in the present analysis. All 48 malignant cases from the participating centers are included in the analytical subset; no cancer-positive examinations were excluded.
Objectives and endpoints The primary objectives are: (1) to demonstrate that the balanced accuracy (BA) of Carebot AI MMG is at least 0.80 in both HSe and HSp operating points; (2) to demonstrate non-inferiority of the AI's balanced accuracy compared with the MRMC "random reader" benchmark with a non-inferiority margin of 0.05; and (3) to demonstrate non-inferiority of sensitivity (Se) of AI versus each of the three radiologists in both HSe and HSp, with a non-inferiority margin of 0.07. Secondary objectives are to describe specificity (Sp), positive predictive value (PPV), negative predictive value (NPV), and to characterize patterns of false-negative and false-positive decisions and their potential implications for clinical risk management.
Statistical analysis Diagnostic performance metrics (Se, Sp, PPV, NPV, BA) are calculated at the case level for Carebot AI MMG and each radiologist in both HSe and HSp. Wilson 95% confidence intervals are used for proportions. Paired McNemar tests are used to compare AI and individual readers in terms of Se and Sp. For balanced accuracy, a MRMC bootstrap procedure with 20,000 iterations is used to construct the distribution of the random reader and to estimate the probability that BA_AI is greater than or equal to BA_random minus the pre-specified margin. Non-inferiority in sensitivity is assessed using a Nam-Blackwelder-type framework on discordant pairs. False-negative and false-positive cases are reviewed qualitatively with emphasis on lesion conspicuity, breast density, and typical error patterns (e.g., dense parenchyma, lesions near the pectoral muscle, benign vascular structures, asymmetries).
Risk, ethics, and data protection The study is non-interventional and entirely retrospective. All mammography examinations were acquired as part of routine care before the study, and all DICOM data were irreversibly anonymized at the site level in compliance with GDPR and applicable national law before transfer to the sponsor. No additional radiation exposure or patient contact occurs, and no adverse events are expected. Given this design, the study does not meet the MDR definition of a clinical investigation under Article 62 and is not subject to prior notification under Article 74(1); individual informed consent is not required. Results are intended to support the clinical evaluation of Carebot AI MMG as a decision-support tool in organized mammography screening.
Inclusion Criteria:
Exclusion Criteria: