Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
This study looks at whether a clinician-facing artificial intelligence (AI) tool can help outpatient doctors spend less time reviewing medical records and documenting care. The tool creates brief summaries of existing medical record information to support routine clinical work. This study examines how use of this tool affects clinicians' workload, time spent in the electronic health record, and overall experience with documentation. The goal is to better understand whether AI documentation support tools can improve efficiency and reduce burden for clinicians in outpatient specialty practice.
Eligible UNC outpatient specialists may be invited to complete surveys and, if they qualify, are randomly assigned either to receive access to the tool (Evidently) at the beginning of the study period or to continue their usual workflow for eight weeks before receiving access. The study does not recruit patients and does not change medical care for patients.
Background and aim: This is a behavioral and health-information-technology study focused on clinical workflow. It does not target a specific patient disease or condition; it evaluates whether a clinician-facing artificial intelligence (AI) tool (Evidently) that produces brief summaries of existing electronic health record information can reduce documentation burden and improve efficiency and experience for outpatient specialists.
Design: Interventional, randomized, parallel-group trial. Eligible clinicians are randomly assigned either to receive access to Evidently at the start of the study period or to continue usual documentation workflow for eight weeks before receiving access. The primary comparison window is eight weeks per arm description in the protocol.
Setting and population: Recruitment targets outpatient attending physicians affiliated with UNC Faculty Practice (UNCFP) or UNC Medical Group (UNCMG) in the United States who can consent and complete study activities in English, with age limits and inclusion of birth-assigned sex categories as specified in the approved IRB application. Participants are recruited by invitation through professional clinical channels; the study does not recruit the general public.
Eligibility (summary): Screening confirms attending physician role; UNCFP or UNCMG affiliation; primary specialty in the protocol-eligible list (for example cardiology, pulmonology, gastroenterology, neurology, oncology, surgery, or anesthesiology); at least two half-day outpatient sessions per week (each at least 3.5 hours); fewer than 25 percent of outpatient sessions jointly staffed with trainees or advanced practice providers at or above the protocol threshold; no participation in a prior Evidently pilot; and no anticipated absence from work longer than one week during the eight-week study period. Individuals who do not meet these criteria are not enrolled.
Procedures for participants: Participation includes an eligibility and baseline survey (approximately 10 to 15 minutes), randomization, use or non-use of Evidently according to assignment during the randomized period, and a brief follow-up survey. There is no monetary compensation. The consent materials describe voluntary participation, confidentiality protections, and that participation is not reported to supervisors or leadership and does not affect employment status, evaluation, or clinical responsibilities.
Outcomes (high level): Outcomes emphasize clinician-reported workload; time and effort related to reviewing and summarizing medical records and using the electronic health record; task-load and related experience measures; and, for those assigned to early access, usability and perceived impact of the tool. The study does not enroll patients and does not assign interventions to patients; any linkage of survey responses to workflow data for analysis follows the approved consent and privacy plan.
Operational identifiers needed to link survey responses to workflow systems, if collected, are described in the approved consent and are handled per institutional policies and the IRB-approved protocol.
Not provided
Not provided
Not provided
Not provided
| Label | Type | Description | Intervention Names |
|---|---|---|---|
| Experimental: Evidently | Experimental | Clinicians randomized to the intervention arm will receive access to a clinician-facing AI clinical summarization tool (Evidently) integrated into routine clinical workflow. The intervention includes standardized onboarding and training materials (e.g., an onboarding webinar or equivalent asynchronous training, tip sheets, and on-demand implementation support). Optional personalization may be available to clinicians but is not required for participation in the study. |
|
| No Intervention: Control | No Intervention | Clinicians randomized to the control arm will continue usual chart review and documentation practices without access to the AI clinical summarization tool during the study period. |
| Name | Type | Description | Arm Group Labels | Other Names |
|---|---|---|---|---|
| AI Clinical Summarization Tool (Evidently) | Behavioral | Clinicians randomized to the intervention arm will receive access to the AI clinical summarization tool following randomization and completion of baseline study procedures. Clinicians randomized to the control arm will continue usual chart review and documentation practices for the duration of the study. Participation in the study and use of the AI clinical summarization tool are voluntary. Clinicians assigned to the intervention arm are not required to use the tool as part of routine care. Use of the AI clinical summarization tool will be characterized using system-generated utilization metrics provided by the vendor. These metrics may include measures such as the number of sessions (i.e., the number of times the tool is invoked), the number of patients for whom the tool is used, and, where available, the number of customized summaries generated. All utilization measures will be aggregated at the clinician level over the study period. |
| Measure | Description | Time Frame |
|---|---|---|
| Change in Clinician Cognitive Load | Cognitive load measured using the Physician Task Load (PTL) scale, adapted from the National Aeronautics and Space Administration (NASA) Task Load Index. The PTL includes four items-mental demand, physical demand, effort, and time pressure-each rated on a 0-100 scale with reference to a typical clinical workday during the past 1-2 weeks. Item scores will be summed to produce a continuous composite score ranging from 0 to 400, with higher scores indicating greater cognitive workload. The primary outcome is the clinician-level change in PTL score from baseline to the end of Study Month 2, analyzed under an intention-to-treat framework. | Baseline to end of Study Month 2 |
| Measure | Description | Time Frame |
|---|---|---|
| Change in time in clinical review | Epic Signal-derived time spent in clinical review, capturing time spent reviewing patient charts and clinical information. | Baseline to Study Month 2 |
| Change in time outside work hours |
Not provided
Inclusion Criteria:
Participants must meet all of the following criteria:
Anesthesiologists will be enrolled under the same protocol but analyzed separately due to distinct workflow characteristics.
Exclusion Criteria:
Clinicians meeting any of the following criteria will be excluded:
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Name | Affiliation | Role |
|---|---|---|
| Spencer Dorn, MD, MPH, MHA | University of North Carolina, Chapel Hill | Principal Investigator |
| Facility | Status | City | State | ZIP | Country | Contacts |
|---|---|---|---|---|---|---|
| UNC Health | Chapel Hill | North Carolina | 27599 | United States |
Deidentified individual participant data that supports the results will be shared beginning 9 to 36 months following publication, provided the investigator who proposes to use the data has approval from an Institutional Review Board (IRB), Independent Ethics Committee (IEC), or Research Ethics Board (REB), as applicable, and executes a data use/sharing agreement with UNC.
Supporting information and deidentified individual participant data will be available beginning 9 months after publication of the primary results and continuing for up to 36 months after publication.
Investigators who wish to access deidentified individual participant data or supporting documents must (1) provide a methodologically sound research proposal, (2) obtain approval from their IRB, IEC, or REB, and (3) execute a data use/sharing agreement with the University of North Carolina at Chapel Hill. Requests should be directed to the study principal investigator. The principal investigator must confirm that the data use agreement has been fully executed and ethics approval has been granted before any data is shared.
Not provided
Not provided
| ID | Term |
|---|---|
| D000077062 | Burnout, Psychological |
| ID | Term |
|---|---|
| D013315 | Stress, Psychological |
| D001526 | Behavioral Symptoms |
| D001519 | Behavior |
Not provided
Not provided
This is a two-arm randomized controlled trial comparing access to a clinician-facing AI clinical summarization tool (Evidently), combined with standardized onboarding and implementation support, versus usual chart review and documentation practices without access to the tool.
Prior to randomization, all eligible clinicians will complete baseline surveys, and six months of retrospective clinician-level electronic health record (EHR) workflow metadata (Epic Signal) will be obtained to characterize baseline workflow and support covariate-constrained randomization. In addition, limited retrospective patient-level data without direct identifiers may be used over the same period to characterize clinician workload and clinical context, including measures such as patients seen per day, visit duration, and aggregated proxies of chart complexity. These patient-level data will be aggregated at the clinician level and will not be used to identify or contact individual patients.
Not provided
Not provided
Clinicians are aware of their assignment. Masking is not feasible given the nature of the intervention, as clinicians will know whether or not they have access to the AI clinical summarization tool. Study outcomes include structured clinician surveys and clinician-level EHR workflow metrics derived from routine clinical operations.
Not provided
|
|
Epic Signal-derived time spent outside scheduled work hours, including evenings and unscheduled days where available.
| Baseline to Study Month 2 |
| Change in clinic preparation time | Self-reported time spent preparing for clinic, during scheduled and outside scheduled work hours (minutes per half-day clinic). | Baseline to Study Month 2 |
| Change in new patient record review time | Self-reported time spent reviewing and summarizing a new patient's record prior to clinic. | Baseline to Study Month 2 |
| Change in burnout | Change in clinician burnout assessed using a validated single-item self-report measure. The item asks clinicians to rate their current level of burnout on a 5-point ordinal scale ranging from 1 (no symptoms of burnout) to 5 (completely burned out). Higher scores indicate greater burnout. The outcome is the clinician-level change in this single-item burnout rating from baseline to the end of Study Month 2. | Baseline to Study Month 2 |
| Number of tool sessions invoked | Total number of times the AI clinical summarization tool (Evidently) was invoked by each clinician during the study period, as recorded by vendor-provided system logs. Reported as a count aggregated at the clinician level (intervention arm only). | Study Month 2 |
| Number of patients with tool use | Total number of unique patients for whom the AI clinical summarization tool (Evidently) was used by each clinician during the study period, as recorded by vendor-provided system logs. Reported as a count aggregated at the clinician level (intervention arm only). | Study Month 2 |
| Proportion of eligible visits with tool use | Proportion of eligible outpatient visits during the study period for which the AI clinical summarization tool (Evidently) was used by each clinician, as derived from vendor-provided system logs and scheduling data. Reported as a percentage aggregated at the clinician level (intervention arm only). | Study Month 2 |
| Context of use | System-generated data describing clinical contexts in which Evidently was used (e.g., new versus return visits, specialty workflows), supplemented by structured survey items where necessary (intervention arm only). | Study Month 2 |
| Usability and fit | Usability and fit are assessed using a study-specific Likert agreement questionnaire comprising three statements about Evidently, each rated on a 5-point Likert agreement scale scored from 1 (Strongly disagree) to 5 (Strongly agree); higher scores indicate more favorable usability and fit. The statements are: Evidently is easy to use; Evidently was easy to learn; Evidently fits the chart review and summarization needs of my specialty. Item-level distributions and/or summary scores (e.g., mean or median per item) will be reported at Study Month 2 (intervention arm only). | Study Month 2 |
| Accuracy and trust | Perceived accuracy and trust are assessed using a study-specific Likert agreement questionnaire comprising four statements, each rated on a 5-point Likert agreement scale scored from 1 (Strongly disagree) to 5 (Strongly agree); higher scores indicate more favorable perceptions of accuracy and trust. The statements address whether Evidently-generated summaries are generally accurate; whether Evidently highlights clinically relevant information; whether Evidently surfaces relevant information the respondent may not have found independently; and whether summaries from Evidently are at least as good as summaries the respondent would otherwise prepare. Item-level distributions and/or summary scores will be reported at Study Month 2 (intervention arm only). | Study Month 2 |
| Impact on day | Single-item measure assessed using a 5-point Likert agreement scale scored from 1 (Strongly disagree) to 5 (Strongly agree); higher scores indicate stronger agreement that Evidently helps the clinician get through their clinical day more easily. The item prompt is: Evidently helps me get through my clinical day more easily. Distribution of responses will be reported at Study Month 2 (intervention arm only). | Study Month 2 |
| Time savings | Self-reported perceived time savings per half-day clinic attributable to Evidently (intervention arm only). | Study Month 2 |
| Impact on care | Single-item measure assessed using a 5-point Likert agreement scale scored from 1 (Strongly disagree) to 5 (Strongly agree); higher scores indicate stronger agreement that Evidently helps the clinician provide better patient care. The item prompt is: Using Evidently helps me provide better patient care. Distribution of responses will be reported at Study Month 2 (intervention arm only). | Study Month 2 |
| Overall satisfaction with AI clinical summarization tool assessed by single-item Likert agreement scale | Overall satisfaction with the AI clinical summarization tool (Evidently) is assessed using a single-item measure on a 5-point Likert agreement scale scored from 1 (Strongly disagree) to 5 (Strongly agree); higher scores indicate greater overall satisfaction. The item prompt is: Overall, I am satisfied with Evidently. The outcome is reported as the distribution of responses (frequency and percentage per category) at Study Month 2 (intervention arm only). | Study Month 2 |
| Open-ended feedback | Qualitative open-ended feedback collected via a single optional free-text item on the end-of-study questionnaire (no numeric scale is used for this measure). The item invites additional narrative feedback about Evidently, including what worked well, where Evidently was most helpful, what was frustrating or less useful than expected, and suggestions for improvement or new functionality. Responses will be analyzed using qualitative content or thematic analysis. Results will be reported as major themes with counts or proportions of respondents whose responses were assigned to each theme where appropriate, plus representative anonymized quotations. If concise enough, a tabulation of the most common paraphrased response categories may be presented; lengthy narrative responses will be summarized thematically rather than as a ranked list of full verbatim answers (intervention arm only). | Study Month 2 |