This randomized quality improvement pilot project aims to assess whether the implementation of generative AI software for documentation, Microsoft Nuance's Digital Ambient eXperience (DAX) Copilot, enhances physician documentation efficiency and reduces burnout.

Background: While electronic health record (EHR) systems have contributed to advances in patient safety and quality of care, they have also been associated with a significant increase in documentation burden, contributing to burnout among clinicians [1]. This is particularly true for physicians with insufficient time for documentation. In some cases, it has resulted in a reduction in appointment slots to allow for additional documentation time, which in turn decreases patient access to care and physician productivity [2,3].

Microsoft Nuance© has recently announced general availability of a new generative artificial intelligence (AI) solution called Digital Ambient eXperience (DAX) DAX CoPilot [4], in which the visit is recorded with patient/parental consent, but a note is generated through the AI along with a visit transcript in near-real time that the provider can use and edit as they see fit. In addition, it allows providers to continue to use their documentation templates while adding the generative AI to "smart sections" within their note. This approach has the potential to substantially reduce documentation burden while maintaining documentation preferences of many providers.

This randomized quality improvement pilot project aims to assess whether the implementation of generative AI software for documentation, Microsoft Nuance's DAX Copilot, can enhance physician documentation efficiency and reduce burnout.

Objectives

Quality Improvement Global Aim: To increase provider documentation efficiency and reduce provider burnout related to documentation burden.

Children's Operational Goal: Determine if the cost of DAX Copilot or related vendor software is justified by reduction in proxies for physician burnout and/or could be offset by seeing more patients in the same time period to improve revenue and patient access.

Goals of the Proposed Work:

- Determine the influence of DAX Copilot on proxies for physician burnout including pajama time, time in notes, and subjective measures of EHR efficiency.

Methods: This is a randomized quality improvement project that will assess changes to proxies for provider documentation efficiency and burnout through a difference-in-differences design.

Project Participants

The project will recruit 20 providers who meet the following inclusion criteria:

1. Practices at Children's in a specialty supported by DAX Copilot product and with available documentation efficiency metrics from Epic's Signal product.

2. >0.5 clinical full time equivalent (cFTE)

3. 2 or more half days per week on average seeing outpatients as the primary provider (not overseeing trainees or APPs).

4. Agrees to use the Children's EHR mobile application (Haiku) on their personal device.

5. Agrees to offer use of the DAX Copilot generative AI software for all patient visit encounters for the duration of the project period

6. Sufficient and stable EHR data on documentation efficiency from Epic's Signal product, defined as:

   1. Having populated data available for at least 40 of the last 52 weeks.
   2. Having stable metrics for pajama time and time in notes in the last 6 months as determined by two physician informaticists based on visual inspection.
   3. >75% of ambulatory documentation completed by the provider themselves (as opposed to taking over a note of a trainee or advanced practice provider).

Participants will be identified based on characteristics such as specialty, cFTE, proportion and location of outpatient work, and Epic's Ambient Opportunity Index from Signal, which is based on normalized scores for proportion of same-day charts closed, pajama type, and characters manually typed.

Randomization will be conducted in blocks of two at the specialty level to ensure equal representation of specialties into one of the two following groups for a 3 month pilot:

1. Use of DAX Copilot generative AI software (AI Group)
2. Continuation of current documentation workflow (Control Group)

Outcomes

Our primary outcomes to be obtained through Epic's Signal product will be:

1. "Pajama Time", defined as the average number of minutes per scheduled day spent in charting activities outside 7 AM to 5:30 PM on weekdays, time outside scheduled hours on weekends, and time on unscheduled holidays. This metric is associated with the exhaustion subscale of the Maslach Burnout Inventory [5].

2. "Time in Notes per Appointment" in minutes

   Additional outcome metrics will include:

3. Progress Note Length (characters)

4. Note Contribution (written by provider vs. others)

5. Time to Appointment Closure

6. Proportion of notes completed using DAX Copilot generative AI software

7. Average patient volume per week

8. Pre- and post-project user responses on a modified KLAS EHR Efficiency and Satisfaction survey, a validated benchmarking tool.

9. Pre- and post-project patient experience scores through routinely capture NRC surveys.

Data Collection

Data will be collected from Epic© Signal and through surveys. The data will include:

• Demographic information of project participants
• Data related to documentation efficiency as outlined in the quality metrics above

Statistical Analysis Our primary analysis will be a difference-in-differences analysis for each outcome. For example, the difference between the provider's average pajama time before and after the intervention period will be calculated for all participants. We will then determine how this average differs in the AI group and in the control group to assess the difference-in-differences.

Additional analyses will include adjusted or stratified difference-in-differences analyses based on provider characteristics listed above. We will also calculate descriptive statistics to compare the outcomes and covariates between the two groups. Depending on the nature of the data, we may use run charts, t-tests, ANOVA, or other appropriate statistical methods to assess the impact of generative AI documentation software.

References

1. Budd, J. Burnout Related to Electronic Health Record Use in Primary Care. J Prim Care Community Health 14, 21501319231166921 (2023).
2. Colicchio, T.K., Cimino, J.J. & Del Fiol, G. Unintended Consequences of Nationwide Electronic Health Record Adoption: Challenges and Opportunities in the Post-Meaningful Use Era. J Med Internet Res 21, e13313 (2019).
3. Gaffney, A., et al. Medical Documentation Burden Among US Office-Based Physicians in 2019: A National Study. JAMA Intern Med 182, 564-566 (2022).
4. Microsoft+Nuance. Automatically document care with Dragon Ambient eXperience (DAX). https://www.nuance.com/asset/en\_us/collateral/healthcare/data-sheet/ds-ambient-clinical-intelligence-en-us.pdf. (2023).
5. Adler-Milstein, J. & Wang, M.D. The impact of transitioning from availability of outside records within electronic health records to integration of local and outside records within electronic health records. J Am Med Inform Assoc 27, 606-612 (2020).

Inclusion Criteria:

Providers:

1. Practices at Children's in a specialty supported by DAX Copilot product and with available documentation efficiency metrics from Epic's Signal product.

2. >0.5 clinical full time equivalent (cFTE)

3. 2 or more half days per week on average seeing outpatients as the primary provider (not overseeing trainees or APPs).

4. Agrees to use the Children's EHR mobile application (Haiku) on their personal device.

5. Agrees to offer use of the DAX Copilot generative AI software for all eligible patient visit encounters for the duration of the project period

6. Sufficient and stable EHR data on documentation efficiency from Epic's Signal product, defined as:

   1. Having populated data available for at least 40 of the last 52 weeks.
   2. Having stable metrics for pajama time and time in notes in the last 6 months as determined by two physician informaticists based on visual inspection.
   3. >75% of ambulatory documentation completed by the provider themselves (as opposed to taking over a note of a trainee or advanced practice provider).

Visits: In person office visits in which interactions occur in English. For example, a visit with an interpreter present who translates verbally into English during the visit would be eligible. A visit in which all interactions occur in another language (i.e. the provider speaks the language the family uses as well) would not be eligible as the AI has not been developed yet for other languages.

Exclusion Criteria:

• Poor WiFi and cellular coverage in the clinic.

Aggregate data will be published. De-identified individual participant data may be requested by other researchers and will be considered on a case-by-case basis.