Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
The purpose of this study is to understand the effects of using an Artificial Intelligence algorithm for skeletal age estimation as a computer-aided diagnosis (CADx) system. In this prospective real-time study, the investigators will send de-identified hand radiographs to the Artificial Intelligence algorithm and surface the output of this algorithm to the radiologist, who will incorporate this information with their normal workflows to make an estimation of the bone age. All radiologists involved in the study will be trained to recognize the surfaced prediction to be the output of the Artificial Intelligence algorithm. The radiologists' diagnosis will be final and considered independent to the output of the algorithm.
The investigators are targeting to study the effect of their Artificial Intelligence algorithm on the radiologists' estimation of skeletal age. Currently, radiologists make the estimation using only the radiographic images and health records. As part of this study, the radiologists will estimate skeletal age from radiographic images, health records, and the output of the CADx algorithm. The investigators wish to understand how radiologists using the Artificial Intelligence algorithm compare to radiologists who do not for the specific task of estimating skeletal age.
This study is organized as a multi-institutional randomized control trial with two arms - experiment (receiving the Artificial Intelligence algorithm's output) and control (no intervention). Both of these arms will be compared to a clinical reference standard ("gold standard") composed of a panel of radiologists. The metric of comparison will be Mean Absolute Distance (MAD). The investigators plan to use statistical tests such as the t-test to determine any statistically-significant difference in skeletal age estimation between the two groups.
The investigators have recruited and analyzed data from a sample size of 1600 exams. Patients getting these exams will not undergo any research procedures that deviate from the current standard practices.
Not provided
Not provided
Not provided
Not provided
| Label | Type | Description | Intervention Names |
|---|---|---|---|
| Control (Without-AI) | No Intervention | This is the control arm where no intervention is provided; represents current standard of care. | |
| Experiment (With-AI) | Experimental | This is the experiment arm where the intervention, "BoneAgeModel", is provided. The participating radiologists in this arm will receive the output of the Artificial Intelligence algorithm. They will be asked to incorporate this new information with their normal workflows to make a diagnosis. The radiologists' diagnosis will be considered final. |
|
| Name | Type | Description | Arm Group Labels | Other Names |
|---|---|---|---|---|
| BoneAgeModel | Device | BoneAgeModel is an Artificial Intelligence tool that takes in a hand radiograph and gender, and outputs the skeletal (bone) age. The intervention involves using this tool as a factor in the clinical decision making process of the participating radiologists. The radiologist's decision will be considered final. |
| Measure | Description | Time Frame |
|---|---|---|
| Paired Difference of Skeletal Age Estimate | Mean absolute difference between dictated final impressions (baseline measure by Radiologist) and the consensus determination of a panel of radiologists following review. | Up to 10 minutes to acquire the scan; up to 2 days to complete diagnosis review |
| Measure | Description | Time Frame |
|---|---|---|
| Time for Diagnosis | Amount of time taken by radiologists when using the BoneAgeModel as compared to when they are not. | Up to approximately 4 minutes |
Not provided
Exams that meet the following inclusion criteria will be included: (1) exams read by radiologists who interpret pediatric skeletal age exams and verbally consent to participate (2) exams that contain a procedure code or study description indicative of a skeletal age exam.
Exams containing more than one radiograph will not be included. Exams for which a trainee provides a preliminary interpretation will be excluded. No further exclusion criteria will be applied on the basis of image quality metrics or manufacturers. No exclusion criteria will be applied on the basis of patient chronological age.
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Name | Affiliation | Role |
|---|---|---|
| Curtis Langlotz, M.D. Ph.D. | Stanford University | Study Chair |
| David Eng, B.S. | Stanford University | Study Director |
| Nishith Khandwala, B.S. | Stanford University | Study Director |
| Safwan Halabi, M.D. | Stanford University | Principal Investigator |
| Facility | Status | City | State | ZIP | Country | Contacts |
|---|---|---|---|---|---|---|
| Stanford University | Stanford | California | 94305 | United States | ||
| Yale New Haven Hospital |
Individual participant data that underlie the results reported in this article after deidentification (text, tables, figures and appendices).
Beginning 3 months and ending 5 years following article publication.
Researchers who provide a methodologically sound proposal.
Not provided
Not provided
Not provided
Not provided
| ID | Title | Description |
|---|---|---|
| FG000 | Control (Without-AI) | Diagnosis by radiologists made according to current standard of care methods. |
| FG001 | Experiment (With-AI) | Diagnosis by radiologists informed by "BoneAgeModel" Artificial Intelligence (AI) algorithm incorporated into normal radiologist workflows and considered as a factor in the clinical decision making process. The radiologists' diagnosis will be considered final. BoneAgeModel takes in a hand radiograph and gender, and outputs the skeletal (bone) age. |
| Title | Milestones | Reasons Not Completed | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall Study |
|
Primary analysis set: Participants with ground-truth labeled exam results and no bone deformity.
Ground-truth labeled: exam was interpreted by a panel of 4 radiologists and their interpretations were averaged to determine a final label.
Not provided
| ID | Title | Description |
|---|---|---|
| BG000 | Control (Without-AI) | Diagnosis by radiologists made according to current standard of care methods. |
| BG001 | Experiment (With-AI) | Diagnosis by radiologists informed by "BoneAgeModel" AI algorithm incorporated into normal radiologist workflows and considered as a factor in the clinical decision making process. |
| Units | Counts |
|---|---|
| Participants |
|
| Title | Description | Population Description | Parameter Type | Dispersion Type | Unit of Measure | Calculate Percentage | Denominator Units Selected | Denominators | Classes |
|---|---|---|---|---|---|---|---|---|---|
| Age, Continuous | Mean |
| Type | Title | Description | Population Description | Reporting Status | Anticipated Posting Date | Parameter Type | Dispersion Type | Unit of Measure | Calculate Percentage | Time Frame | Units Analyzed | Denominator Units Selected | Arm/Group Information | Denominators | Classes | Analyses | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Primary | Paired Difference of Skeletal Age Estimate | Mean absolute difference between dictated final impressions (baseline measure by Radiologist) and the consensus determination of a panel of radiologists following review. | Primary analysis set: Participants with ground-truth labeled exam results and no bone deformity. Ground-truth labeled: exam was interpreted by a panel of 4 radiologists and their interpretations were averaged to determine a final label. | Posted | Mean | 95% Confidence Interval | months | Up to 10 minutes to acquire the scan; up to 2 days to complete diagnosis review |
|
Day of study visit (average approximately 1 hour)
Not provided
Not provided
| ID | Title | Description | Deaths (Affected) | Deaths (At Risk) | Serious Events (Affected) | Serious Events (At Risk) | Other Events (Affected) | Other Events (At Risk) |
|---|---|---|---|---|---|---|---|---|
| EG000 | Control (Without-AI) | Diagnosis by radiologists made according to current standard of care methods. |
Not provided
Not provided
Not provided
| Title | Organization | Phone | Extension | |
|---|---|---|---|---|
| Safwan S Halabi, MD | Stanford University | (650) 721-2850 | safwan.halabi@stanford.edu |
Not provided
| Type | Includes Protocol | Includes SAP | Includes ICF | Document Label | Document Date | Document Uploaded Date | Document File Name |
|---|---|---|---|---|---|---|---|
| Prot_SAP | Yes | Yes | No | Study Protocol and Statistical Analysis Plan | Aug 1, 2018 | Sep 30, 2020 | Prot_SAP_000.pdf |
Not provided
A hand radiograph will be randomly assigned to one of two groups - control and experiment. In the control group, participating radiologists will diagnose the exam using the current standard of care (no intervention). In the experiment group, the radiologists will factor in the output of the Artificial Intelligence algorithm in their skeletal age estimation. In all cases, the decision of the radiologist will be considered final.
Not provided
Not provided
Not provided
Not provided
|
| New Haven |
| Connecticut |
| 06519 |
| United States |
| Boston Children's Hospital | Boston | Massachusetts | 02115 | United States |
| New York University | New York | New York | 10016 | United States |
| Cincinnati Children's Hospital Medical Center | Cincinnati | Ohio | 45229 | United States |
| Children's Hospital of Philadelphia | Philadelphia | Pennsylvania | 19104 | United States |
| BG002 | Total | Total of all reporting groups |
| years |
|
| Age, Customized | Count of Participants | Participants |
|
| Sex: Female, Male | Count of Participants | Participants |
|
| Race and Ethnicity Not Collected | Race and Ethnicity were not collected from any participant. | Count of Participants | Participants |
|
| Region of Enrollment | Number | participants |
|
| Skeletal age final impression (mean) | Final impression is the baseline radiologist assessment. | Mean | Standard Deviation | years |
|
| Skeletal age final impression (categorical) | Final impression is the baseline radiologist assessment. | Count of Participants | Participants |
|
| Clinical histories | Clinical history types | Count of Participants | Participants |
|
Diagnosis by radiologists informed by "BoneAgeModel" AI algorithm incorporated into normal radiologist workflows and considered as a factor in the clinical decision making process. |
|
|
|
| Secondary | Time for Diagnosis | Amount of time taken by radiologists when using the BoneAgeModel as compared to when they are not. | Primary analysis set: Participants with ground-truth labeled exam results and no bone deformity. Ground-truth labeled: exam was interpreted by a panel of 4 radiologists and their interpretations were averaged to determine a final label. | Posted | Median | Inter-Quartile Range | seconds | Up to approximately 4 minutes |
|
|
|
|
| 0 |
| 939 |
| 0 |
| 939 |
| 0 |
| 939 |
| EG001 | Experiment (With-AI) | Diagnosis by radiologists informed by "BoneAgeModel" AI algorithm incorporated into normal radiologist workflows and considered as a factor in the clinical decision making process. | 0 | 964 | 0 | 964 | 0 | 964 |
Not provided
Not provided