DYNAMIC AI: Digital Innovation With Remote Management and Predictive Modelling to Integrate COPD Care With Artificial Intelligence-based Insights: An Acceptability, Feasibility and Safety Study

Digital Innovation With Remote Management and Predictive Modelling to Integrate COPD Care With Artificial Intelligence-based Insights: An Acceptability, Feasibility and Safety Study

DYNAMIC AI is an MHRA-regulated medical device trial that will examine the feasibility of using AI predictive models within COPD multi-disciplinary team meetings, to allow clinicians to prioritise and optimise COPD management.

Background: COPD need, challenge and opportunity.

Chronic obstructive pulmonary disease (COPD) is a global healthcare challenge. Whilst care-quality gaps exist across the continuum of patient's with COPD's presentation, COPD exacerbations are responsible for a large proportion of the disease-burden, adverse outcomes and healthcare costs. People with COPD prioritise the avoidance of exacerbations and resultant hospitalisations. The Patient Charter for COPD (March 2021), called for proactive, preventative management to reduce the risk of exacerbations and premature death1. Digital transformation with co-designed digital tools and emerging innovations such as wearable sensors and AI-based predictive modelling offers the opportunity to address these COPD care-quality gaps and achieve this care re-orientation.

DYNAMIC: NHS GG&C COPD digital service transformation, implementation and evidence.

'DYNAMIC' (Digital Innovation with Remote Management and Predictive Modelling to Integrate COPD Care) program commenced in 2018. This was based on an innovation partnership between NHS GG&C respiratory medicine and West of Scotland innovation teams and with LenusHealth (then StormID). Initial funding was from a digital health technology catalyst award from InnovateUK. Subsequent Scottish Government Technology Enabled Care (TEC) Program funding has continued the service and allowed scale-up provision.

Our vision was to initiate digital transformation of a COPD service with the development, implementation, and evaluation of co-designed digital tools. We co-designed the "LenusCOPD' patient and clinician web applications and support website based on cycles of user experience testing. We aimed to establish tools which would be sustainably used by patients during follow-up, and support COPD co-management with an anticipation of achieving reduction in respiratory-related admissions and occupied bed days in a high-risk COPD cohort. Evidence was gathered in the 'RECEIVER' observational cohort study and subsequently in the scale-up 'DYNAMIC-SCOT' service evaluation.

Key results from RECEIVER / DYNAMIC-SCOT2:

• Participants continue to use the LenusCOPD patient app, with an average of 3.5 interactions per person per week sustained >1-year post-onboarding.
• COPD-related hospital admissions and occupied bed days were reduced following LenusCOPD onboarding in participants with a history of a severe exacerbation in the previous year, with a median time to readmission of 380 days compared with 50 days in a contemporary matched control patient cohort.
• Median time to admission or death was 43 days in control, 338 days in RECEIVER and 400 days in DYNAMIC-SCOT participants with a severe exacerbation in the preceding year.
• The 12-month risk of admission or death was 74% in control patients, 53% in RECEIVER and 47% in the DYNAMIC-SCOT sub-cohort participants.

These feasibility and utility results support scale-up adoption of these digital tools. Scale-up provision of the core LenusCOPD service will continue in NHS GG&C 2022-2024, supported by renewed Scottish Government TEC funding. Currently ~400 patients have been onboarded the COPD digital service (March 2022), with user numbers anticipated to rise as LenusCOPD is used to support recovery and re-orientation of COPD diagnostic, regular review and pulmonary rehabilitation services. This continued adoption and our innovation partnership provides the test-bed infrastructure to implement and evaluate additional innovations, including the artificial intelligence-insights for MDT decision support proposed in this clinical investigation.

Rationale: Trained and validated COPD risk prediction models

The challenges outlined in above presented an opportunity to use AI to develop patient risk prediction models. The data collected as part of the service, together with historical Electronic Health Record data were used to create a suite of risk prediction models. The following steps were taken to develop each model.

• Data exploration - data exploration and visualisations to identify features, biases, and other issues with the datasets.
• Problem formulation - how to define the target variable of interest, and what modelling approach to use.
• Data splitting - defining the strategy for splitting the datasets to be used for training, validation, and testing.
• Data cleaning and missing data handling - identifying erroneous data, outliers in the dataset, and defining a strategy for dealing with missing data (imputation, exclusion, allowing missing values).
• Feature Engineering - The process of transforming and aggregating the raw data into features to be used for model training. Feature engineering allows for greater insight to be drawn from the data.
• Model training - exploration of different machine learning algorithms and choosing the best candidate for the problem at hand. The processed features are then used for model training.
• Model validation - validation and reporting on model performance using the hold-out test dataset.
• Model explainability - use of third-party methods to make the model predictions as interpretable and explainable as possible.
• Model Fairness - examining model performance on sub-groups of interest and potential model re-training to account for any inequalities.

For full details of model development, refer to the investigator brochure and Model approval report. The output for each of the models is a number representative of the probability of the event of interest happening for each patient.

Intervention

We have co-designed all of the components required to provide live AI-model based risk scores for 12-month mortality and 3-month readmission to clinicians within a COPD-MDT. To ensure the production COPD digital service is not impacted by the AI Insights study, a separate clinician dashboard will be developed to surface these model predictions. The output of the exacerbation model will be a 'check-in' message sent to patients, highlighting that they have some possible changes in their data, prompting them to review their self-management plan and/or contact clinical team for advice if required. This is summarised in table 1, with the example check-in message at figure 1.

All the source data feeds required to run these models have been established in NHS GG&C Azure tenancy, using Azure data science VMs (Virtual Machine). The intervention consists of: -

Consent process

Further details are in CIP section 10.4 and in the "Standard operating procedure for DYNAMIC-AI consent" document. The LenusCOPD digital service support website https://support.nhscopd.scot provides the detailed patient information and links to explanation videos which support the consent process. Study information would not be live on this open access website until all study approvals were obtained. The web content is available for review prior to approvals at https://support.test.nhscopd.scot/dynamic-ai

The LenusCOPD patient app contains the consent screens where invited patients can view information about the study and accept, reject or defer consent and access further information via the support website, or request discussion with the study team. The LenusCOPD clinician dashboard AI insights consent page contains the user interface for activating consent in the patient app, revoking consent when requested by patient and auditing consent status.

AI insights data pipelines These cloud-based digital connections ingest and aggregate relevant health record data (demographics, diagnoses, admissions, prescribing, laboratory results) from consented participants. Data is aggregated, pseudonymised and converted into suitable formats for use by the AI insights service. Processes for providing this data are approved and supported by NHS GG&C SafeHaven team, with a standard operating procedure.

AI insights model serving pipeline The cloud-based digital pipeline ingests the pre-processed data from the data pipeline for use by the machine-learning models, producing model predictions and explainability for consenting participants for use by the AI Insights Clinician App.

AI insights service This consists of cloud-based electronic health system back-end functions and processes which receive instructions and perform actions to ensure the smooth running of the intervention.

AI insights database The data store for the AI Insights digital service. Raw and summary data can be exported as model performance reports for review by data science and clinical leads, to clinical trials unit for independent analysis, and for long term retention (as per IRAS document).

AI insights clinician app The web application that provides clinicians with clinical dashboards (Figure 1 & 2) to view and export patient lists and review machine-learning model results, explainability and fairness data from consented participants.

STUDY OBJECTIVES

Hypotheses

Patients with COPD will find it acceptable and consent to use of their routine healthcare data to provide AI insights risk prediction scores to their clinical team, with data used for ongoing model training and validation.

It will be technically feasible and safe to provide AI-model derived 12-month mortality COPD risk prediction scores for review in the COPD AI-MDT within the 'AI insights' application.

Primary objectives

To determine :

1. the acceptability to patients with COPD and
2. technical feasibility and
3. safety

of presenting live artificial intelligence-based 12-month mortality risk-prediction score from the LenusCOPD AI insights application to COPD clinicians' multi-disciplinary team meetings in NHS GG&C.

Primary feasibility, acceptability and safety measurements

These will be reported descriptively as there is no pilot data to inform a power calculation or a priori success metric.

1. Evaluation of acceptability will be based on proportion of patients invited who consent to participate in the DYNAMIC-AI study.
2. Evaluation of technical feasibility will be proportion of participants with adequate source data in LenusCOPD who have 12-month mortality model-risk scores calculated and presented for MDT review in the AI-insights model app.
3. Evaluation of safety will be based on occurrence of device-related adverse events and from the prospective evaluation of model risk scores - actions of clinicians based on model risk scores and calibration of predicted events : occurred events.

Secondary objectives

Acquire detailed acceptability and technical feasibility experience with the 12-month mortality risk prediction model.

Expand dataset for training and validation of 3-month respiratory-related admission and COPD exacerbation risk prediction models.

Evaluate technical feasibility (proportion of consented participants with live scores in model app) and descriptive adoption experience with 3-month respiratory-related admission and COPD exacerbation models, if review of validation performance data indicates these are ready for clinical adoption.

Establish a preliminary experience and utility dataset from use of live AI insights by a COPD multidisciplinary team.

Secondary exploratory evaluations

Detailed acceptability data

Proportion of invited patients who reviewed additional supporting information (video and written materials).

Proportion of invited patients who require individual clinician input to inform their consent decision, with themes from these discussions noted including any divergence on acceptability/consent for the study and consent to use data for model training.

Detailed technical feasibility and model performance data

Captured in model performance reports. Serial data across the clinical investigation including:

• numbers of participants with missing model scores and reasons (missing data per feature group, latency of data refresh, participant withdrawal, others).
• model performance metrics calculated using the hold-out test dataset for each model run including AUC-ROC, precision-recall AUC, classification report, and confusion matrix.
• Model fairness metrics calculated for groups of interest.
• Global and local explainability for the model to illustrate model decision making and bio-plausibility
• Performance of model risk prediction vs events captured in LenusCOPD (death at 12 months, respiratory-related hospital admission, exacerbation).
• drift-variability in model performance with exploratory evaluation of reasons (e.g. missing data resolved, change in underlying source data, patient condition fluctuates) for any identified unexpected changes in model performance or individual risk scores.

Expand dataset for training and validation of 3-month respiratory-related admission and COPD exacerbation risk prediction models

Data will be accrued from consenting participants. Updated model performance reports will be reviewed by study steering committee. If these and interim analyses with primary endpoint are satisfactory these models will be added to the live service.

Preliminary experience and utility data from MDT use of live COPD AI insights Number of patients discussed at COPD AI-MDT.

Clinical actions taken from model risk score review in MDT, captured in LenusCOPD events and notes.

If exacerbation model adopted for live clinical use:

• Number of check-in messages triggered by high-risk score from exacerbation model - per patient and total for cohort captured in LenusCOPD messaging.
• Proportion of participants where exacerbation model check-in messages are toggled off temporarily or permanently, captured in LenusCOPD notes.
• Outcomes from check-in messages - patient initiates contact, other event, captured in LenusCOPD messaging and notes.

Patient events (death, hospital admission, COPD exacerbation; captured in LenusCOPD) as they occur prospectively following model risk score. Noting intervening MDT review, messaging, clinician action, patient action triggered by model to establish preliminary experience data about impact of risk-prediction triggered actions and their potential influence on the subsequent accuracy of those predictions. Evaluation will compare the model performance vs prospective events considering patients who had a model triggered review/action vs those who did not. This evaluation will be supplemented by external retrospective validation data acquired from adoption of the LenusCOPD management service in other health boards. The model performance on the groups with and without intervention will also be shared with the Data Science team to allow for further model refinements and re-training.

Related evaluations Continued development of AI insights models up to the point of commencement of this investigation is being undertaken within the DYNAMIC project and RECEIVER trial evaluations. External validation of the AI insights' model performance will be undertaken using de-identified electronic health record and LenusCOPD service data from organisations other than NHS GG&C. Co-design, clinical user experience and acceptability testing of AI insights will continue in parallel. These separate evaluations are supported by all appropriate approvals, governance and documentation.

INVESTIGATION PLAN

Study Design

This clinical investigation will be performed according to the UK Policy Framework for Health and Social Care Research (2017).

This is an observational cohort implementation-effectiveness study. Interim effectiveness evaluations will be undertaken at 3-monthly intervals across the trial, with reporting on the primary endpoints reviewed by the study steering committee. Results from these interim evaluations may trigger a recommendation to adapt the implementation or intervention - e.g. changes in consent information, technical updates. Proposed adaptations to the implementation strategy or intervention would be considered by the study management committee, with any required amendments to this CIP and associated study documentation submitted for sponsor and REC consideration. Implementation changes would only be made once all reviews were complete and approvals obtained. The final report of the study would detail any changes made based on the interim evaluations.

Inclusion Criteria:

Current active user of NHS Scotland COPD digital service. Resident and under care of NHS GG&C COPD multidisciplinary team. Consent, access to technology (smartphone, tablet, computer and daily internet access), ability to understand written English and confirmed diagnosis of COPD are requirements for use of the COPD digital service

-

Exclusion Criteria:

Lack of provision of informed consent.

-