Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Name | Class |
|---|---|
| Beth Israel Deaconess Medical Center | OTHER |
| University of Minnesota | OTHER |
Not provided
Not provided
Not provided
Not provided
This study will evaluate the effect of providing access to GPT-4, a large language model, compared to traditional management decision support tools on performance on case-based management reasoning tasks.
Artificial intelligence (AI) technologies, specifically advanced large language models like OpenAI's ChatGPT, have the potential to improve medical decision-making. Although ChatGPT-4 was not developed for its use in medical-specific applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultation. However, little is known about how ChatGPT augments the clinical reasoning abilities of clinicians.
Clinical reasoning is a complex process involving pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could potentially help reduce clinician workload and decrease the likelihood of mismanagement. However, ChatGPT-4 was not developed for clinical reasoning nor has it been validated for this purpose. Further, it may be subject to disinformation, including convincing confabulations that may mislead clinicians. If clinicians misuse this tool, it may not improve reasoning and could even cause harm. Therefore, it is important to study how clinicians use large language models to augment clinical reasoning prior to routine incorporation into patient care.
In this study, participants will be randomized to answer clinical management cases with or without access to ChatGPT-4. Each case has multiple components, and the participants will be asked to discuss their reasoning for each component. Answers will be graded by independent reviewers blinded to treatment assignment. A grading rubric was developed for each case by a panel of 4-7 expert discussants. Discussants independently developed a rubric for each case, and then any discrepancies were resolved through multiple rounds of discussions.
Not provided
Not provided
Not provided
Not provided
| Label | Type | Description | Intervention Names |
|---|---|---|---|
| GPT-4 | Active Comparator | Group will be given access to GPT-4 |
|
| Usual Resources | No Intervention | Group will not be given access to GPT-4 but will be encouraged to use any resources they wish besides large language models (UpToDate, Dynamed, google, etc). |
| Name | Type | Description | Arm Group Labels | Other Names |
|---|---|---|---|---|
| GPT-4 | Other | OpenAI's GPT-4 large language model with chat interface. |
|
| Measure | Description | Time Frame |
|---|---|---|
| Management Reasoning | Percent correct (range: 0 to 100) for each case. | Within one-hour study |
| Measure | Description | Time Frame |
|---|---|---|
| Time Spent on Management | Time (in minutes) participants spend per case between the two study arms. | Within one-hour study |
Not provided
Inclusion Criteria:
Exclusion Criteria:
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Name | Affiliation | Role |
|---|---|---|
| Jonathan H Chen, MD, PhD | Stanford University | Principal Investigator |
| Adam Rodman, MD | Beth Israel Deaconess Medical Center | Principal Investigator |
| Andrew Olson, MD | University of Minnesota | Principal Investigator |
| Facility | Status | City | State | ZIP | Country | Contacts |
|---|---|---|---|---|---|---|
| Stanford University | Palo Alto | California | 94304 | United States |
| PubMed Identifier | Type | Citation | Retractions |
|---|---|---|---|
| 39910272 | Derived | Goh E, Gallo RJ, Strong E, Weng Y, Kerman H, Freed JA, Cool JA, Kanjee Z, Lane KP, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Hom J, Chen JH, Rodman A. GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nat Med. 2025 Apr;31(4):1233-1238. doi: 10.1038/s41591-024-03456-y. Epub 2025 Feb 5. | |
| 39148822 |
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
The trial will be designed as a randomized, two-arm, single-blind parallel group study.
Not provided
Not provided
The grading of responses will be performed by assessors blinded to participant identity and treatment assignment.
| Goh E, Gallo R, Strong E, Weng Y, Kerman H, Freed J, Cool JA, Kanjee Z, Lane KP, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Hom J, Chen JH, Rodman A. Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial. medRxiv [Preprint]. 2024 Aug 7:2024.08.05.24311485. doi: 10.1101/2024.08.05.24311485. |