Not provided
| ID | Type | Description | Link |
|---|---|---|---|
| 1K23DK125718-01A1 | U.S. NIH Grant/Contract | View source |
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Name | Class |
|---|---|
| National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) | NIH |
Not provided
Not provided
Not provided
Not provided
Not provided
The purpose of this research study is to measure the effect on of a large language model interface on the usability, attitudes, and provider trust when using a machine learning algorithm-based clinical decision support system in the setting of bleeding from the upper gastrointestinal tract (upper GIB). Specifically, the investigators are looking to assess the optimal implementation of such machine learning algorithms in simulation scenarios to best engender trust and improve usability. Participants will be randomized to either machine learning algorithm alone or algorithm with a large language model interface and exposed to simulation cases of upper GIB.
The experiment will deploy a previously validated machine learning algorithm trained on existing clinical datasets within simulation scenarios in which a patient with acute gastrointestinal bleeding (at low, moderate, and high risk for poor outcome) is evaluated.
Prior to the simulation, a baseline educational module about artificial intelligence, machine learning, and clinical decision support will be provided to all participants. The investigators will establish psychological safety by detailing what is available in the room, the opportunity to call a consultant, and availability of laboratory and radiographic studies. Each clinical scenario will run for approximately 10 minutes based on real patient cases where vital signs change over time and laboratory values are made available at specific points in the assessment. The study will evaluate the effect of a large language model-based interaction with the machine learning algorithm with interpretability dashboard compared to the machine learning algorithm with interpretability dashboard alone. Each participant will receive three scenarios in randomized order of risk.
For the large language model interaction arm, participants will be provided the computer workstation a LLM chatbot interface of the algorithm and interpretability dashboard For the machine learning dashboard arm, participants will be provided the computer workstation with the algorithm and interpretability dashboard.
Not provided
Not provided
Not provided
Not provided
| Label | Type | Description | Intervention Names |
|---|---|---|---|
| Large Language Model-based Interaction | Experimental | LLM-powered chatbot with the machine learning dashboard to provide the risk assessment and provide rationale based on interpretability metrics provided by the dashboard in which study participants can directly interact with using natural language. Participants will be provided the Generative Pre-trained Transformer (GPT) chatbot powered machine learning model dashboard. |
|
| Machine Learning Dashboard | No Intervention | Machine learning algorithm output with an interactive dashboard that can be used to explain, or interpret the input factors that contribute most towards the generated risk score. Participants will have access to the machine learning dashboard only. |
| Name | Type | Description | Arm Group Labels | Other Names |
|---|---|---|---|---|
| LLM | Other | Use of a Large Language Model (LLM) chatbot interface to Interact with the Machine Learning Algorithm and interpretability dashboard. |
|
| Measure | Description | Time Frame |
|---|---|---|
| Median Change in Attitudes Towards Machine Learning Algorithms in Clinical Care Using UTAUT | The study will use a common set of dependent variables to assess baseline and post-intervention attitudes towards machine learning algorithms in clinical care using an adapted Unified Theory of Acceptance and Use of Technology (UTAUT) survey assessing perceived usefulness of the system, perceived ease of use, attitudes towards using it, behavioral intentions, and trust, measured with a 5-point Likert scale. Percent change in UTAUT survey response between Large Language Model-based Interaction and Machine Learning Dashboard at recruitment prior to administration of scenarios and immediately after completion of scenarios. The difference in time between the two will be approximately 60 minutes. Higher change indicates greater acceptance/intention to use the GutGPT+Dashboard. | Approximately 60 minutes |
| Measure | Description | Time Frame |
|---|---|---|
| Clinician Decision Making of Triage of GI Bleeding | Mean percentage of decision accuracy per participant. Accuracy is defined as the percentage of times participants accurately choose the correct clinical decision for each simulation scenario of acute upper GI bleeding for each treatment condition. Immediately after completion of scenarios (60 minutes from initiation of study for each participant). No further follow up afterwards. |
Not provided
Inclusion Criteria:
Exclusion Criteria:
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Name | Affiliation | Role |
|---|---|---|
| Dennis Shung, MD | Yale School of Medicine Section of Digestive Diseases | Principal Investigator |
| Facility | Status | City | State | ZIP | Country | Contacts |
|---|---|---|---|---|---|---|
| Yale New Haven Hospital | New Haven | Connecticut | 06510 | United States |
| PubMed Identifier | Type | Citation | Retractions |
|---|---|---|---|
| 27521511 | Background | Laine L. Risk Assessment Tools for Gastrointestinal Bleeding. Clin Gastroenterol Hepatol. 2016 Nov;14(11):1571-1573. doi: 10.1016/j.cgh.2016.08.003. Epub 2016 Aug 10. No abstract available. | |
| 22310222 | Background | Laine L, Jensen DM. Management of patients with ulcer bleeding. Am J Gastroenterol. 2012 Mar;107(3):345-60; quiz 361. doi: 10.1038/ajg.2011.480. Epub 2012 Feb 7. |
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| ID | Title | Description |
|---|---|---|
| FG000 | Large Language Model-based Interaction | LLM-powered chatbot with the machine learning dashboard to provide the risk assessment and provide rationale based on interpretability metrics provided by the dashboard in which study participants can directly interact with using natural language. Participants will be provided the Generative Pre-trained Transformer (GPT) chatbot powered machine learning model dashboard. |
| FG001 | Machine Learning Dashboard | Machine learning algorithm output with an interactive dashboard that can be used to explain, or interpret the input factors that contribute most towards the generated risk score. Participants will have access to the machine learning dashboard only. |
| Title | Milestones | Reasons Not Completed | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall Study |
|
|
Baseline demographics presented here are for completers.
Not provided
| ID | Title | Description |
|---|---|---|
| BG000 | Large Language Model-based Interaction | LLM-powered chatbot with the machine learning dashboard to provide the risk assessment and provide rationale based on interpretability metrics provided by the dashboard in which study participants can directly interact with using natural language. Participants will be provided the Generative Pre-trained Transformer (GPT) chatbot powered machine learning model dashboard. |
| Units | Counts |
|---|---|
| Participants |
|
| Title | Description | Population Description | Parameter Type | Dispersion Type | Unit of Measure | Calculate Percentage | Denominator Units Selected | Denominators | Classes |
|---|---|---|---|---|---|---|---|---|---|
| Age, Customized | Count of Participants |
| Type | Title | Description | Population Description | Reporting Status | Anticipated Posting Date | Parameter Type | Dispersion Type | Unit of Measure | Calculate Percentage | Time Frame | Units Analyzed | Denominator Units Selected | Arm/Group Information | Denominators | Classes | Analyses | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Primary | Median Change in Attitudes Towards Machine Learning Algorithms in Clinical Care Using UTAUT | The study will use a common set of dependent variables to assess baseline and post-intervention attitudes towards machine learning algorithms in clinical care using an adapted Unified Theory of Acceptance and Use of Technology (UTAUT) survey assessing perceived usefulness of the system, perceived ease of use, attitudes towards using it, behavioral intentions, and trust, measured with a 5-point Likert scale. Percent change in UTAUT survey response between Large Language Model-based Interaction and Machine Learning Dashboard at recruitment prior to administration of scenarios and immediately after completion of scenarios. The difference in time between the two will be approximately 60 minutes. Higher change indicates greater acceptance/intention to use the GutGPT+Dashboard. | Posted | Median | 95% Confidence Interval | units on a scale | Approximately 60 minutes |
|
60 minutes
Not provided
Not provided
| ID | Title | Description | Deaths (Affected) | Deaths (At Risk) | Serious Events (Affected) | Serious Events (At Risk) | Other Events (Affected) | Other Events (At Risk) |
|---|---|---|---|---|---|---|---|---|
| EG000 | Large Language Model-based Interaction | LLM-powered chatbot with the machine learning dashboard to provide the risk assessment and provide rationale based on interpretability metrics provided by the dashboard in which study participants can directly interact with using natural language. Participants will be provided the Generative Pre-trained Transformer (GPT) chatbot powered machine learning model dashboard. LLM: Use of a Large Language Model (LLM) chatbot interface to Interact with the Machine Learning Algorithm and interpretability dashboard. |
Not provided
Not provided
Not provided
| Title | Organization | Phone | Extension | |
|---|---|---|---|---|
| Sunny Chung | Yale School of Medicine | 203-824-1459 | Sunny.chung@yale.edu |
Not provided
| Type | Includes Protocol | Includes SAP | Includes ICF | Document Label | Document Date | Document Uploaded Date | Document File Name |
|---|---|---|---|---|---|---|---|
| Prot | Yes | No | No | Study Protocol | Jan 1, 2023 | Dec 16, 2025 | Prot_000.pdf |
| SAP | No | Yes | No | Statistical Analysis Plan | Nov 11, 2023 | Dec 16, 2025 | SAP_001.pdf |
| ICF | No | No | Yes | Informed Consent Form | Feb 25, 2023 | Dec 16, 2025 | ICF_002.pdf |
Not provided
| ID | Term |
|---|---|
| D006471 | Gastrointestinal Hemorrhage |
| ID | Term |
|---|---|
| D005767 | Gastrointestinal Diseases |
| D004066 | Digestive System Diseases |
| D006470 | Hemorrhage |
| D010335 | Pathologic Processes |
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Approximately 60 minutes |
| Background | Leonardi, P. M. 2009. Why do people reject new technologies and stymie organizational changes of which they are in favor? Exploring misalignments between social interactions and materiality. Human Communication Research, 35(3): 407-441. |
| Background | Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User Acceptance of Information Technology: Toward a Unified View. MIS Quarterly, 27(3), 425-478 |
| 40825997 | Derived | Chung S, Giuffre M, Rajashekar N, Pu Y, Shin YE, Kresevic S, Chan C, Nakamura-Sakai S, You K, Saarinen T, Hsiao A, Wong AH, Evans L, McCall T, Kizilcec RF, Sekhon J, Laine L, Shung DL. Usability and adoption in a randomized trial of GutGPT a GenAI tool for gastrointestinal bleeding. NPJ Digit Med. 2025 Aug 18;8(1):527. doi: 10.1038/s41746-025-01896-5. |
| BG001 | Machine Learning Dashboard | Machine learning algorithm output with an interactive dashboard that can be used to explain, or interpret the input factors that contribute most towards the generated risk score. Participants will have access to the machine learning dashboard only. |
| BG002 | Total | Total of all reporting groups |
| Participants |
|
| Sex: Female, Male | Count of Participants | Participants | No |
|
| Race (NIH/OMB) | Count of Participants | Participants | No |
|
| Ethnicity (NIH/OMB) | Not collected | Count of Participants | Participants | No |
|
| Training level | Count of Participants | Participants | No |
|
| Familiarity with Artificial Intelligence (AI) | Count of Participants | Participants |
|
| Mean baseline Unified Theory of Acceptance and Use of Technology (UTAUT) survey score | Total score range for each scale 1-5. Higher scores generally indicating higher acceptance | Mean | Standard Deviation | score on a scale |
|
LLM-powered chatbot with the machine learning dashboard to provide the risk assessment and provide rationale based on interpretability metrics provided by the dashboard in which study participants can directly interact with using natural language. Participants will be provided the Generative Pre-trained Transformer (GPT) chatbot powered machine learning model dashboard. |
| OG001 | Machine Learning Dashboard | Machine learning algorithm output with an interactive dashboard that can be used to explain, or interpret the input factors that contribute most towards the generated risk score. Participants will have access to the machine learning dashboard only. |
|
|
| Secondary | Clinician Decision Making of Triage of GI Bleeding | Mean percentage of decision accuracy per participant. Accuracy is defined as the percentage of times participants accurately choose the correct clinical decision for each simulation scenario of acute upper GI bleeding for each treatment condition. Immediately after completion of scenarios (60 minutes from initiation of study for each participant). No further follow up afterwards. | Posted | Mean | Standard Deviation | percent accuracy per participant | Approximately 60 minutes |
|
|
|
| 0 |
| 52 |
| 0 |
| 52 |
| 0 |
| 52 |
| EG001 | Machine Learning Dashboard | Machine learning algorithm output with an interactive dashboard that can be used to explain, or interpret the input factors that contribute most towards the generated risk score. Participants will have access to the machine learning dashboard only. | 0 | 56 | 0 | 56 | 0 | 56 |
Not provided
Not provided
Not provided
| D013568 |
| Pathological Conditions, Signs and Symptoms |
| Native Hawaiian or Other Pacific Islander |
|
| Black or African American |
|
| White |
|
| More than one race |
|
| Unknown or Not Reported |
|