Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Name | Class |
|---|---|
| University of Oxford | OTHER |
| Imperial College London | OTHER |
Not provided
Not provided
Not provided
Not provided
This project will apply AI technology to meet the gap between increasing demand and limited capacity of high- volume healthcare services. The project will develop evidence that will support the safe deployment of Ufonia's automated telemedicine platform to deliver calls to cataract surgery patients at two large NHS hospital trusts.
The proposed study will implement DORA in addition to the current standard of care for a cohort of patients at Imperial College Healthcare Trust and Oxford University Hospitals NHS Foundation Trust. The study will evaluate the agreement of DORA's decision with an expert clinician. In addition it will test the acceptability of the solution for patients and clinicians; the sensitivity and specificity of the system in deciding if a patient requires additional review; and the health economic benefits of the solution to patients (reduced time and travel) and the local healthcare system. If successful, a proposal will be developed to roll the solution out to all patients at each site in anticipation of an application to a late phase award for wider NHS deployment.
Background
Due to an ageing population and increased expectation, the demand for many services is exceeding the capacity of the clinical workforce. As a result, staff are facing a crisis of burnout from being pressured to deliver high- volume workloads, driving increasing costs for providers. Artificial intelligence, in the form of conversational agents, presents a possible opportunity to enable efficiencies in the delivery of care.
Aims and Objectives
This study aims to evaluate the effectiveness, usability and acceptability of DORA - an AI-enabled autonomous telemedicine call - for detection of post-operative cataract surgery patients who require further assessment. The study's objectives are: to establish efficacy of DORA's decision making in comparison to an expert human clinician; baseline sensitivity and specificity for detection of true complications; evaluation of patient acceptability; evidence for cost-effectiveness; and to capture data that may support further studies.
Project plan and methods used
Based on implementation science, the interdisciplinary study will be a mixed-methods phase one pilot establishing inter-observer reliability; as well as usability and acceptability.
Timelines for delivery
The study will last eighteen months: seven months of evaluation and intervention refinement, nine months of implementation and follow-up, and two months of post-evaluation analysis and write-up.
Anticipated Impact and Dissemination
The project's key contributions will be evidence on artificial intelligence voice conversational agent effectiveness, and associated usability and acceptability. Results will be disseminated in peer-reviewed journals and at international medical sciences and engineering conferences.
Not provided
Not provided
Not provided
Not provided
Not provided
| Label | Type | Description | Intervention Names |
|---|---|---|---|
| Dora follow-up phone call | Experimental | DORA uses a variety of AI technologies to deliver the patient follow-up call, including: speech transcription, natural language understanding, a machine-learning conversation model to enable contextual conversations, and speech generation. Together, these technologies cover the input, processing and analysis, and output needed to maintain a natural conversation. DORA is configured to deliver calls through a telephone connection as a real-time, stand-alone system: the operator inputs individual patient details to initiate the call and completes a summary in the electronic health record (EHR) afterwards. The entire conversation will be supervised by a clinician. This clinician will be able to interrupt the call at any point if the system fails, the patient struggles to interact with it, or DORA does not collect sufficient information from the patient. The clinician will record a clinical assessment which will be compared to the DORA assessment. |
|
| Name | Type | Description | Arm Group Labels | Other Names |
|---|---|---|---|---|
| Dora | Other | DORA uses a variety of AI technologies to deliver the patient follow-up call, including: speech transcription, natural language understanding, a machine-learning conversation model to enable contextual conversations, and speech generation. Together, these technologies cover the input, processing and analysis, and output needed to maintain a natural conversation. DORA is configured to deliver calls through a telephone connection as a real-time, stand-alone system: the operator inputs individual patient details to initiate the call and completes a summary in the electronic health record (EHR) afterwards. |
| Measure | Description | Time Frame |
|---|---|---|
| Agreement | Inter-rater reliability: the degree of agreement between DORA and the clinician on their assessments of the individual symptoms and the management plan; Whether or not the clinician had to interrupt the call to ask clarifying questions | Inter-rater reliability was assessed based on data collected during Dora calls, which lasted an average of 7.5 minutes |
| Measure | Description | Time Frame |
|---|---|---|
| Clinical Complications Identified or Missed by DORA System | Clinical data was collected from patients' electronic health record (EHR) up to 90 days postoperatively to capture numbers of participants 'recommended discharge' by Dora R1 with subsequent unexpected management change | Up to 90 days post surgery |
Not provided
Inclusion Criteria:
Exclusion Criteria:
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Name | Affiliation | Role |
|---|---|---|
| Eduardo Normando, MD, PhD | Imperial College London | Study Chair |
| Facility | Status | City | State | ZIP | Country | Contacts |
|---|---|---|---|---|---|---|
| Imperial College Healthcare NHS Trust | London | United Kingdom | ||||
| Oxford University Hospitals NHS Foundation Trust |
| PubMed Identifier | Type | Citation | Retractions |
|---|---|---|---|
| 33090118 | Background | Milne-Ives M, de Cock C, Lim E, Shehadeh MH, de Pennington N, Mole G, Normando E, Meinert E. The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review. J Med Internet Res. 2020 Oct 22;22(10):e20346. doi: 10.2196/20346. | |
| 34319248 | Background | de Pennington N, Mole G, Lim E, Milne-Ives M, Normando E, Xue K, Meinert E. Safety and Acceptability of a Natural Language Artificial Intelligence Assistant to Deliver Clinical Follow-up to Cataract Surgery Patients: Proposal. JMIR Res Protoc. 2021 Jul 28;10(7):e27227. doi: 10.2196/27227. |
Not provided
Not provided
Not provided
Not provided
Not provided
| ID | Title | Description |
|---|---|---|
| FG000 | Dora Follow-up Phone Call | DORA uses a variety of AI technologies to deliver the patient follow-up call, including: speech transcription, natural language understanding, a machine-learning conversation model to enable contextual conversations, and speech generation. DORA is configured to deliver calls through a telephone connection as a real-time, stand-alone system: the operator inputs individual patient details to initiate the call and completes a summary in the electronic health record (EHR) afterwards. The entire conversation will be supervised by a clinician. This clinician will be able to interrupt the call at any point if the system fails, the patient struggles to interact with it, or DORA does not collect sufficient information from the patient. The clinician will record a clinical assessment which will be compared to the DORA assessment. Dora: DORA uses a variety of AI technologies to deliver the patient follow-up call, including: speech transcription, natural language understanding, a machine-learning conversation model to enable contextual conversations, and speech generation. Together, these technologies cover the input, processing and analysis, and output needed to maintain a natural conversation. DORA is configured to deliver calls through a telephone connection as a real-time, stand-alone system: the operator inputs individual patient details to initiate the call and completes a summary in the electronic health record (EHR) afterwards. |
| Title | Milestones | Reasons Not Completed | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall Study |
|
|
Not provided
Not provided
| ID | Title | Description |
|---|---|---|
| BG000 | Dora Follow-up Phone Call | DORA uses a variety of AI technologies to deliver the patient follow-up call, including: speech transcription, natural language understanding, a machine-learning conversation model to enable contextual conversations, and speech generation. DORA is configured to deliver calls through a telephone connection as a real-time, stand-alone system: the operator inputs individual patient details to initiate the call and completes a summary in the electronic health record (EHR) afterwards. The entire conversation will be supervised by a clinician. This clinician will be able to interrupt the call at any point if the system fails, the patient struggles to interact with it, or DORA does not collect sufficient information from the patient. The clinician will record a clinical assessment which will be compared to the DORA assessment. Dora: DORA uses a variety of AI technologies to deliver the patient follow-up call, including: speech transcription, natural language understanding, a machine-learning conversation model to enable contextual conversations, and speech generation. Together, these technologies cover the input, processing and analysis, and output needed to maintain a natural conversation. DORA is configured to deliver calls through a telephone connection as a real-time, stand-alone system: the operator inputs individual patient details to initiate the call and completes a summary in the electronic health record (EHR) afterwards. |
| Units | Counts |
|---|---|
| Participants |
|
| Title | Description | Population Description | Parameter Type | Dispersion Type | Unit of Measure | Calculate Percentage | Denominator Units Selected | Denominators | Classes |
|---|---|---|---|---|---|---|---|---|---|
| Age, Continuous | Mean |
| Type | Title | Description | Population Description | Reporting Status | Anticipated Posting Date | Parameter Type | Dispersion Type | Unit of Measure | Calculate Percentage | Time Frame | Units Analyzed | Denominator Units Selected | Arm/Group Information | Denominators | Classes | Analyses | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Primary | Agreement | Inter-rater reliability: the degree of agreement between DORA and the clinician on their assessments of the individual symptoms and the management plan; Whether or not the clinician had to interrupt the call to ask clarifying questions | Calls to 7 of the 202 were incomplete. | Posted | Count of Participants | Participants | Inter-rater reliability was assessed based on data collected during Dora calls, which lasted an average of 7.5 minutes |
|
Up to 3 months post cataract surgery
Not provided
Not provided
| ID | Title | Description | Deaths (Affected) | Deaths (At Risk) | Serious Events (Affected) | Serious Events (At Risk) | Other Events (Affected) | Other Events (At Risk) |
|---|---|---|---|---|---|---|---|---|
| EG000 | Dora Follow-up Phone Call | DORA uses a variety of AI technologies to deliver the patient follow-up call, including: speech transcription, natural language understanding, a machine-learning conversation model to enable contextual conversations, and speech generation. DORA is configured to deliver calls through a telephone connection as a real-time, stand-alone system: the operator inputs individual patient details to initiate the call and completes a summary in the electronic health record (EHR) afterwards. The entire conversation will be supervised by a clinician. This clinician will be able to interrupt the call at any point if the system fails, the patient struggles to interact with it, or DORA does not collect sufficient information from the patient. The clinician will record a clinical assessment which will be compared to the DORA assessment. Dora: DORA uses a variety of AI technologies to deliver the patient follow-up call, including: speech transcription, natural language understanding, a machine-learning conversation model to enable contextual conversations, and speech generation. Together, these technologies cover the input, processing and analysis, and output needed to maintain a natural conversation. DORA is configured to deliver calls through a telephone connection as a real-time, stand-alone system: the operator inputs individual patient details to initiate the call and completes a summary in the electronic health record (EHR) afterwards. |
Not provided
| Term | Organ System | Source Vocabulary | Assessment Type | Notes | Statistical Information |
|---|---|---|---|---|---|
| Symptomatic patients with unexpected management changes | Eye disorders | Systematic Assessment | For patients who 'passed' their phone assessment, but were found to be symptomatic at follow-up with unexpected management changes apart from lubricants or advice, such as surgical intervention and additional specialist follow-up or investigations |
Not provided
| Title | Organization | Phone | Extension | |
|---|---|---|---|---|
| Prof Edward Meinert | Newcastle University | 191 208 6000 | edward.meinert@newcastle.ac.uk |
Not provided
| Type | Includes Protocol | Includes SAP | Includes ICF | Document Label | Document Date | Document Uploaded Date | Document File Name |
|---|---|---|---|---|---|---|---|
| Prot | Yes | No | No | Study Protocol | May 17, 2021 | May 14, 2024 | Prot_000.pdf |
| SAP | No | Yes | No | Statistical Analysis Plan | Aug 30, 2022 | May 14, 2024 | SAP_001.pdf |
Not provided
| ID | Term |
|---|---|
| D002386 | Cataract |
| D058442 | Capsule Opacification |
| ID | Term |
|---|---|
| D007905 | Lens Diseases |
| D005128 | Eye Diseases |
Not provided
Not provided
The study will be a multi-centre, mixed-methods clinical investigation to develop evidence regarding the feasibility, acceptability and potential effectiveness of Dora.
Not provided
Not provided
No masking is possible, because all participants receive the same intervention.
Not provided
|
| Calls Completed Without Intervention |
Number of autonomous calls that were completed without needing any intervention from the supervising clinician; Clinician-reported reasons for asking clarifying questions |
| Dora calls lasted an average of 7.5 minutes |
| System Usability | Measured using the System Usability Scale (minimum of 0, maximum of 100, higher scores indicate better usability) | Usability assessments were completed up to 6 months after the Dora call |
| Usability of Telehealth System Implementation | Measured using the Telehealth Usability Questionnaire (minimum score of 1, maximum score of 5, averaged across 19 items; higher scores indicate better usability) | Usability was assessed up to 6 months after the call |
| Qualitative Patient Perspectives of Usability | Qualitative feedback from semi-structured interviews | Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call |
| Acceptability of AI Follow-up Phone Call | Qualitative feedback from semi-structured interviews | Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call |
| Satisfaction With AI Follow-up Phone Call | Qualitative feedback from semi-structured interviews | Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call |
| Appropriateness of AI for Follow-up Assessment | Qualitative feedback from semi-structured interviews | Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call |
| Cost Impact | A cost analysis compared the direct costs of face-to-face (F2F) follow-up at Imperial with Dora R1 (in Oxford, patients do not have routine postoperative follow-up). Assumptions included annual costs for various healthcare professionals and the duration of F2F follow-up appointments (estimated at 30 min). | Conducted 6 months after baseline |
| Subsequent Unplanned Follow-up | Clinical data was collected from patients' electronic health record (EHR) up to 90 days postoperatively to capture numbers of participants 'recommended discharge' by Dora R1 with subsequent unplanned review. | Up to 90 days post surgery |
| Oxford |
| United Kingdom |
| 40503087 | Derived | Milne-Ives M, Homer SR, Andrade J, Meinert E. Mapping the Process of Engagement With Digital Health Interventions: A Cross-Case Synthesis. Mayo Clin Proc Innov Qual Outcomes. 2025 May 27;9(3):100625. doi: 10.1016/j.mayocpiqo.2025.100625. eCollection 2025 Jun. |
| 39050586 | Derived | Meinert E, Milne-Ives M, Lim E, Higham A, Boege S, de Pennington N, Bajre M, Mole G, Normando E, Xue K. Accuracy and safety of an autonomous artificial intelligence clinical assistant conducting telemedicine follow-up assessment for cataract surgery. EClinicalMedicine. 2024 Jul 3;73:102692. doi: 10.1016/j.eclinm.2024.102692. eCollection 2024 Jul. |
| years |
|
| Sex: Female, Male | Count of Participants | Participants |
|
| Race/Ethnicity, Customized | Ethnicity data was not available for 2 participants. | Count of Participants | Participants |
|
| First or second cataract surgery | Count of Participants | Participants |
|
| Income | Not all participants completed the self-report questionnaire or provided data for every item | Count of Participants | Participants |
|
| Education level | Not all participants completed the self-report questionnaire or provided data for every item | Count of Participants | Participants |
|
The supervising clincian's decision about whether the participant could be discharged after their call without requiring further clinician assessment.
|
|
| Secondary | Clinical Complications Identified or Missed by DORA System | Clinical data was collected from patients' electronic health record (EHR) up to 90 days postoperatively to capture numbers of participants 'recommended discharge' by Dora R1 with subsequent unexpected management change | Posted | Count of Participants | Participants | Up to 90 days post surgery |
|
|
|
| Secondary | Calls Completed Without Intervention | Number of autonomous calls that were completed without needing any intervention from the supervising clinician; Clinician-reported reasons for asking clarifying questions | Posted | Number | DORA follow-up calls | Dora calls lasted an average of 7.5 minutes |
|
|
|
| Secondary | System Usability | Measured using the System Usability Scale (minimum of 0, maximum of 100, higher scores indicate better usability) | Not all participants completed the usability questionnaire | Posted | Mean | Standard Deviation | units on a scale | Usability assessments were completed up to 6 months after the Dora call |
|
|
|
| Secondary | Usability of Telehealth System Implementation | Measured using the Telehealth Usability Questionnaire (minimum score of 1, maximum score of 5, averaged across 19 items; higher scores indicate better usability) | Not all participants completed the usability questionnaire | Posted | Mean | Standard Deviation | units on a scale | Usability was assessed up to 6 months after the call |
|
|
|
| Secondary | Qualitative Patient Perspectives of Usability | Qualitative feedback from semi-structured interviews | Participants invited for semi-structured interviews - categories reflect major components of the Theoretical Framework of Acceptability and the number of participants who provided feedback that was coded into those categories. | Posted | Number | participants | Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call |
|
|
|
| Secondary | Acceptability of AI Follow-up Phone Call | Qualitative feedback from semi-structured interviews | Participants invited for semi-structured interviews - categories reflect major components of the Theoretical Framework of Acceptability and the number of participants who provided feedback that was coded into those categories. | Posted | Number | participants | Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call |
|
|
|
| Secondary | Satisfaction With AI Follow-up Phone Call | Qualitative feedback from semi-structured interviews | Interviewed participants were asked if they would be willing to use Dora again; their responses were coded into the three categories below. | Posted | Number | participants | Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call |
|
|
|
| Secondary | Appropriateness of AI for Follow-up Assessment | Qualitative feedback from semi-structured interviews | Participants invited for semi-structured interviews - categories reflect major components of the Theoretical Framework of Acceptability and the number of participants who provided feedback that was coded into those categories. | Posted | Number | participants | Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call |
|
|
|
| Secondary | Cost Impact | A cost analysis compared the direct costs of face-to-face (F2F) follow-up at Imperial with Dora R1 (in Oxford, patients do not have routine postoperative follow-up). Assumptions included annual costs for various healthcare professionals and the duration of F2F follow-up appointments (estimated at 30 min). | Patients on the Imperial post-cataract surgery pathway | Posted | Number | Great British Pounds | Conducted 6 months after baseline |
|
|
|
| Secondary | Subsequent Unplanned Follow-up | Clinical data was collected from patients' electronic health record (EHR) up to 90 days postoperatively to capture numbers of participants 'recommended discharge' by Dora R1 with subsequent unplanned review. | Posted | Count of Participants | Participants | Up to 90 days post surgery |
|
|
|
| 0 |
| 202 |
| 0 |
| 202 |
| 4 |
| 202 |
|
Not provided
Not provided
| Title | Measurements |
|---|---|
|
| Intervention coherence |
|
| Title | Measurements |
|---|---|
|
| Title | Measurements |
|---|---|
|
| Title | Measurements |
|---|---|
|
|