Large Language Model for Understanding and Monitoring Elderly Neurocognition

LUMEN (Large Language Model for Understanding and Monitoring Elderly Neurocognition): A Clinical Feasibility Study With Nested Qualitative Evaluation for AI- Assisted Dementia Assessment

Dementia affects millions of people worldwide, and early diagnosis is essential for getting the right care and support. Doctors rely on collateral histories (accounts from family members or caregivers) to understand changes in a person's memory and thinking. However, these histories can be incomplete, unstructured, or difficult to obtain, making diagnosis more challenging.

This study will test LUMEN (Large Language Model for Understanding and Monitoring Elderly Neurocognition), an AI-powered conversation tool designed to help caregivers describe their loved one's symptoms more effectively. By asking structured questions and guiding the conversation, LUMEN can create clear, well-organised reports for memory clinic doctors. This could make assessments quicker, more accurate, and less stressful for families.

We will test LUMEN in real-world clinics by asking caregivers and doctors to use it and provide feedback. We want to understand how easy it is to use, whether it could improve the quality of information shared, and how it fits into existing NHS memory clinic processes. We will also run co-production workshops with community groups to ensure the tool is accessible to people from diverse cultural and language backgrounds.

This research is exciting because it explores how artificial intelligence can improve dementia care. If successful, LUMEN could enhance the diagnostic process, reduce carer burden, and help more people access dementia support sooner. In the future, this tool could be used nationwide in memory clinics, improving care for thousands of families.

1. Background and Rationale Dementia diagnosis relies on clinical history, cognitive assessments, and collateral information from caregivers. However, obtaining structured and reliable collateral histories is challenging due to caregiver burden, recall bias, and time constraints in memory clinics. Large language models (LLMs) have the potential to improve this process by guiding structured history-taking, extracting clinically relevant details, and standardising information gathering.

   LUMEN (Large Language Model for Understanding and Monitoring Elderly Neurocognition) is an AI-powered conversational tool designed to assist caregivers in providing structured collateral histories before clinical dementia assessments. Seed-funded by the Royal College of Psychiatrists, the Alzheimer's Research UK Northern Network and the Newcastle Biomedical Research Network, LUMEN was developed to enhance diagnostic accuracy, reduce caregiver burden, and improve workflow efficiency in memory clinics.

   This study will evaluate the feasibility, usability, and acceptability of LUMEN in real-world clinical settings through a nested qualitative feasibility study. We will recruit caregivers and clinicians to interact with LUMEN and assess its effectiveness in gathering collateral histories. The study will also incorporate co-production workshops with community groups to explore issues around language, cultural inclusivity, and accessibility. Findings will inform future refinements and a larger validation study.

2. Research Objectives

This study aims to:

1. Evaluate LUMEN's usability -Measure caregiver and clinician experiences using validated scales (System Usability Scale [SUS] and NASA Task Load Index [NASA-TLX]).
2. Explore acceptability and implementation barriers and facilitators-Conduct qualitative interviews and coproduction workshops to refine LUMEN's design, ensuring accessibility across diverse user groups.
3. Compare LUMEN's outputs to clinician assessments-Investigate the alignment between LUMEN-generated reports and clinician-elicited histories using Cohen's Kappa for inter-rater reliability.

3. Background Work Undertaken LUMEN was co-developed with clinicians and stakeholders using a Modified Delphi process involving 150 clinicians, primarily old age psychiatrists. This produced 2,600 questions mapped to validated dementia tools, ensuring clinical relevance. Early technical testing evaluated open-source LLMs (e.g., Mistral 7B, LLaMA2 7B) using simulated clinical vignettes. ROC curve analysis demonstrated an AUC of 0.89 for distinguishing dementia-related narratives, and Cohen's Kappa 0.82 indicated high inter-rater reliability between LUMEN's outputs and clinician ratings. Early PPIE (Patient and Public Involvement and Engagement) feedback highlighted the need for simpler language, cultural sensitivity, and improved user experience. These insights will be further refined through the proposed co-production workshops.

4. Study Design and Methodology This study will employ a mixed-methods approach, combining quantitative usability testing with qualitative interviews to evaluate LUMEN's feasibility and acceptability.

4.1. Participants and Recruitment

• Carers (n=20-30 dyads): Recruited via Northumbria NHS Memory Clinics. Eligible carers will be adults (≥18 years) who know the patient well, have basic English literacy, and can provide informed consent.
• Patients (n=20-30): Individuals (≥65 years) attending a memory clinic for cognitive assessment.
• Clinicians (n=8-10): Specialists in dementia care (neurologists, psychiatrists, geriatricians, advanced nurse practitioners) with ≥2 years of experience.

4.2. Study Procedures

1. Baseline Data Collection:

   • Demographics (age, gender, ethnicity, socioeconomic status).
   • Clinical diagnosis (where available).
   • Cognitive test scores (MoCA/ACE).

2. LUMEN Interaction:

   • Carers will use LUMEN on a laptop or tablet to provide structured collateral histories (~20-30 min).
   • Clinicians will review LUMEN-generated histories to assess completeness and clinical utility.

3. Usability and Cognitive Load Assessment:

   • SUS (System Usability Scale): 10-item Likert-scale questionnaire assessing ease of use (score ≥70 = good usability).
   • NASA-TLX (Task Load Index): Evaluates perceived cognitive workload (scores 0-29 = low workload).

4. Qualitative Evaluation:

   • Semi-structured interviews (n=10 carers, n=4-5 clinicians).
   • Co-production workshops with community groups to explore language, cultural representation, and accessibility.

5. Exploratory Analysis:

   • Inter-rater reliability (Cohen's Kappa): Agreement between LUMEN outputs and clinician assessments.
   • Diagnostic accuracy (AUROC curve analysis): Comparison of LUMEN's assessments with clinical diagnoses.

5. Milestones and Timescales

The study will run for 12 months (April 2025 - April 2026), with key phases as follows:

Phase Month Activities Preparation & Ethics 1-3 Final IRAS approval, NHS R&D approvals, team setup Recruitment & Data Collection 4-9 Patient-carer dyad recruitment, clinician recruitment, LUMEN testing Usability Workshops 6-8 Co-production workshops, qualitative interviews Data Analysis & Refinement 9-10 Thematic analysis, SUS/NASA-TLX scoring, LLM refinements Final Reporting & Next Steps 11-12 Dissemination, grant preparation for large-scale validation study 6. Expected Impact This study will generate critical feasibility data to inform a larger clinical validation study and subsequent NHS integration. Expected outcomes include:

• A refined, co-designed LUMEN prototype with improved usability and accessibility.
• Quantitative evidence on usability and cognitive workload, supporting future funding applications.
• Preliminary insights into AI-assisted collateral history-taking, laying the foundation for regulatory approval as a Software as a Medical Device (SaMD).
• Potential for early NHS adoption under the MHRA framework, streamlining dementia diagnostics.

Carers of patients attending Memory Clinic appointments; Aged 18 or over; Both patients and cares can provide informed consent; Basic English literacy

Clinicians must have at least 2 years of specialist dementia care experience and be personally responsible for dementia diagnosis; Must be provide informed consent