Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
This study evaluates how well anonymized artificial-intelligence (AI) tools perform on standardized pediatric case vignettes and whether showing AI suggestions can improve clinicians' answers. About 30 board-certified/eligible pediatric specialists at a single hospital complete a one-time session. Participants are randomized to two groups. Group A (n≈15): physicians answer each vignette once. Group B (n≈15): physicians answer and rate confidence (1-10), then review anonymized suggestions from five different AI tools (tool names not shown) and may keep or change their answer; changes and confidence are recorded.
Primary focus: measure AI performance (diagnostic accuracy, medication-dosing accuracy, interpretation accuracy) overall and by difficulty tier, and record AI response time. Secondary focus: quantify how AI suggestions affect human performance (change in accuracy, direction of change, confidence shift, and time). No patients or biospecimens are involved; risks are minimal (time and possible discomfort with performance review). Findings may inform safe, evidence-based ways to use AI alongside clinicians in pediatrics.
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Label | Type | Description | Intervention Names |
|---|---|---|---|
| Group/Cohort 1: Direct Answer (Physician-only) | Board-certified/eligible pediatric specialists answer each standardized vignette once, without confidence scoring and without viewing AI suggestions. Outcomes captured: Diagnostic accuracy, dosing accuracy, interpretation accuracy, completion time. | ||
| Group/Cohort 2: Confidence + AI Suggestions | Pediatric specialists first answer and rate confidence on a 1-10 scale; then they view anonymized suggestions from five distinct AI tools (tool names not shown) and may keep or revise their answer. All changes and confidence shifts are recorded. Outcomes captured: Pre- vs post-AI accuracy (and direction of change), dosing and interpretation accuracy changes, confidence shift, completion time. |
|
| Name | Type | Description | Arm Group Labels | Other Names |
|---|---|---|---|---|
| AI Suggestions (Anonymized 5-tool panel) | Other | What: Display of AI-generated suggestions for each vignette, aggregated from five large language model tools (names not shown to participants). When/Who: Shown only in Group 2, after the physician's initial answer and confidence score. Purpose: Measure AI performance (primary) and quantify the effect of AI suggestions on physicians' answers (secondary). Applies to: Group 2. |
| Measure | Description | Time Frame |
|---|---|---|
| AI Interpretation Accuracy (%) | Proportion of correct laboratory/imaging interpretations or appropriate next-test selections, per AI tool and pooled; stratified by difficulty tier. Unit: percent (0-100). | Day 1 |
| AI Diagnostic Accuracy (%) | Proportion of vignettes with a correct primary diagnosis produced by each anonymized AI tool and pooled across tools. Correctness is defined against a pre-specified reference answer key; results are also stratified by pre-defined difficulty tiers (easy/moderate/difficult/very difficult). Unit of measure: percent (0-100). | Day 1 |
| AI Medication-Dosing Accuracy (%) | Proportion of dose recommendations meeting pediatric standards (weight- or BSA-based ranges, route, frequency) per reference rubric, per AI tool and pooled; stratified by difficulty tier. Unit: percent (0-100). | Day 1 |
| Measure | Description | Time Frame |
|---|---|---|
| Change in Physician Diagnostic Accuracy (percentage points) (Group 2 only) | Post-AI accuracy minus pre-AI accuracy per participant on the same case set; also categorized as beneficial (incorrect→correct), harmful (correct→incorrect), or no change. Accuracy is the proportion of cases with a correct final diagnosis according to a prespecified answer key. | Day 1: Baseline (pre-AI) and immediate Post-AI within the same session (0-15 min after baseline). |
Not provided
Inclusion Criteria:
Exclusion Criteria:
Not provided
Not provided
Board-certified/eligible general pediatrics specialists recruited from SBÜ Sultangazi Haseki Training and Research Hospital network.
Not provided
Not provided
| Facility | Status | City | State | ZIP | Country | Contacts |
|---|---|---|---|---|---|---|
| SBÜ Sultangazi Haseki Training and Research Hospital | Istanbul | Sultangazi | 34010 | Turkey (Türkiye) |
| PubMed Identifier | Type | Citation | Retractions |
|---|---|---|---|
| 40489764 | Background | Su H, Sun Y, Li R, Zhang A, Yang Y, Xiao F, Duan Z, Chen J, Hu Q, Yang T, Xu B, Zhang Q, Zhao J, Li Y, Li H. Large Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis. J Med Internet Res. 2025 Jun 9;27:e72062. doi: 10.2196/72062. | |
| 39504445 | Background | Bicknell BT, Butler D, Whalen S, Ricks J, Dixon CJ, Clark AB, Spaedy O, Skelton A, Edupuganti N, Dzubinski L, Tate H, Dyess G, Lindeman B, Lehmann LS. ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis. JMIR Med Educ. 2024 Nov 6;10:e63430. doi: 10.2196/63430. |
Not provided
Not provided
IPD Description:
No IPD will be shared. The dataset comprises detailed, item-level responses from a small, single-center cohort of pediatric specialists. Despite de-identification, the risk of re-identification is non-trivial given granular performance metrics and professional identifiers. The informed consent did not include permission to share raw individual responses outside the study team.
Plan to Share Supporting Materials: Yes Supporting Documents: Study Protocol, Statistical Analysis Plan, scoring rubrics, redacted vignette templates, and analysis code.
Time Frame: Available within 6 months after the primary manuscript is published and for at least 36 months thereafter.
Access Criteria and URL: Materials will be provided upon reasonable request to the Principal Investigator (email: drberkerokay@gmail.com). A data use agreement will be required; use is limited to non-commercial research and aggregate reporting.
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
|
| Confidence Rating Task (1-10 Likert) | Other | What: Self-rated confidence for the initial answer on a 1-10 scale. When/Who: Group 2 before viewing AI suggestions. Purpose: Quantify confidence changes pre- vs post-AI and relate confidence to correctness. Applies to: Group 2. |
|
| Confidence Shift (Δ on a 1-10 scale) (Group 2 only) | Post-AI self-rated confidence minus pre-AI confidence; association with correctness is examined. Unit: scale points (-9 to +9). | Day 1: Baseline (pre-AI) and immediate Post-AI within the same session (0-15 min after baseline). |
| Answer-Change Frequency (%) (Group 2 only) | Proportion of vignettes for which physicians revised their initial answer after AI suggestions; reported overall and by difficulty tier. Unit: percent (0-100). | Day 1 |
| AI Response Time (seconds per vignette) | Time from vignette display to final AI output, reported per tool and pooled; also by difficulty tier. Unit: seconds. | Day 1 |
| Net Benefit Index of AI Exposure (percentage points) (Group 2 only) | Beneficial change rate (incorrect→correct) minus harmful change rate (correct→incorrect) for diagnostic items; sensitivity analyses for dosing/interpretation. Unit: percentage points. | Day 1 |
| 39509461 | Background | Cross JL, Choma MA, Onofrey JA. Bias in medical AI: Implications for clinical decision-making. PLOS Digit Health. 2024 Nov 7;3(11):e0000651. doi: 10.1371/journal.pdig.0000651. eCollection 2024 Nov. |
| 32908284 | Background | Cruz Rivera S, Liu X, Chan AW, Denniston AK, Calvert MJ; SPIRIT-AI and CONSORT-AI Working Group; SPIRIT-AI and CONSORT-AI Steering Group; SPIRIT-AI and CONSORT-AI Consensus Group. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nat Med. 2020 Sep;26(9):1351-1363. doi: 10.1038/s41591-020-1037-7. Epub 2020 Sep 9. |
| 35584845 | Background | Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, Denniston AK, Faes L, Geerts B, Ibrahim M, Liu X, Mateen BA, Mathur P, McCradden MD, Morgan L, Ordish J, Rogers C, Saria S, Ting DSW, Watkinson P, Weber W, Wheatstone P, McCulloch P; DECIDE-AI expert group. Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. BMJ. 2022 May 18;377:e070904. doi: 10.1136/bmj-2022-070904. |