Development and Evaluation of the Electronic Frailty Index+ (eFI+) Tool: Integrated Prognostic-decision Modelling to Target Interventions for Older People With Moderate or Severe Frailty
Development and Evaluation of the Electronic Frailty Index+ (eFI+) Tool: Integrated Prognostic-decision Modelling to Target Interventions for Older People With Moderate or Severe Frailty
Research questions
i) How should electronic frailty index (eFI) components be combined with additional routine primary care data to develop prognostic models for predicting key outcomes of requirement for home care, falls/fractures, nursing home admission and mortality in older people with moderate or severe frailty?
ii) Can model predictive performance be improved through addition of data from measures that are practical for primary care use, but not available in routine data?
iii) How should risk predictions from the prognostic models be translated into a decision analytic model (DAM) to guide clinical management?
iv) What is the potential cost-effectiveness of implementing interventions targeted at subgroups of older people with frailty in routine NHS care?
Background
Lead applicant Clegg led the eFI development, validation and national implementation. This has been translated into major UK health policy change through inclusion in the 2017/18 GP contract, which supports frailty stratification using the eFI, and UK National Health Service Long Term Plan.
Aim
To develop and evaluate the eFI+, a prognostic tool supplementing the original eFI including 4 integrated prognostic-decision models. The eFI+ will stratify older people with moderate or severe frailty into subgroups most likely to benefit from key interventions (community rehabilitation; falls prevention; comprehensive geriatric assessment; advance care planning).
Methods
Design
Prognostic model development, internal validation and external validation using large datasets (ResearchOne, SAIL databank, Leeds Data Model) and cohort study data (CARE75+), with linked DAM and health economic analysis.
Population
Patients ≥65 with moderate or severe frailty, defined by the existing eFI.
Key outcomes
12-month outcomes for prognostic models:
Statistical methods
i) Prognostic modelling
The investigators will build 4 separate prognostic models for our 4 key outcomes by combining the eFI with additional individual-level routine data, informed by reviews to identify prognostic factors. Each model will be developed and internally validated in one large dataset, to adjust for potential overfitting, with subsequent external validation of predictive performance in a second large dataset.
Separately, the investigators will use CARE75+ (n≈1,200) to investigate additional predictive value of clinical measures practical for primary care (e.g. gait speed, activities of daily living, loneliness).
ii) Decision analytic model (DAM)
The investigators will translate the prognostic models into a framework to support clinical decision-making, in co-production with stakeholders/PPI. The investigators will integrate prognostic models with effect size estimates from systematic reviews/meta-analyses to identify relevant thresholds of predicted risk, above which implementation of our key interventions would be warranted.
iii) Health economic evaluation
12-month and long-term cost effectiveness models will be developed, informed by the DAM.
Health technologies being assessed
The eFI+ will be developed using components of the original eFI, supplemented with additional routine primary care EHR data, and guidance on the added benefits of implementing simple clinical measures in routine primary care practice. The eFI+ will be suitable for rapid implementation in UK primary care EHR systems, building on existing close links with system suppliers (SystmOne/EMISWeb/Vision/Microtest).
The investigators will develop, then internally and externally validate the eFI+ using the Secure Anonymised Information Linkage (SAIL) databank, the ResearchOne database, and the Leeds Data Model (LDM).
In addition, the investigators will analyse Community Ageing Research 75+ (CARE75+) cohort study data (CI Clegg, n≈1,200) as the only national cohort study to include eFI scores, to investigate how simple measures that can be assessed in primary care, but are not available in routine EHR data (e.g. gait speed, timed-up-and-go test; activities of daily living; loneliness) may improve prediction.
Study design
Prognostic model development, internal validation and external validation using routine primary care research data (ResearchOne), linked datasets (SAIL databank and LDM) and cohort study data (CARE75+), with integrated Decision Analytic Modelling, including health economic analysis.
Databases
Secure Anonymised Information Linkage (SAIL) databank
Anonymised records from around 5 million people in Wales, with linked primary care, ED attendance, hospital admissions, outpatient data, social care, Welsh Care Homes Dataset, and ONS mortality data. SAIL includes eFI summary scores and individual components.
ResearchOne
Nationally representative, de-identified data from around 6 million UK primary care electronic health records on the TPP SystmOne clinical system. ResearchOne includes eFI summary scores and individual components.
Leeds Data Model (LDM)
Anonymised, linked primary, secondary, community and social care data from 810,000 patients across 108 practices in Leeds, including eFI summary scores and individual components.
Community Ageing Research 75+ (CARE75+) cohort
National prospective cohort study (n≈1,200) collecting detailed sociodemographic information, frailty measures (including eFI scores), simple instruments suitable for use in primary care (e.g. gait speed, timed-up-and-go test; activities of daily living; informal care; loneliness), and key outcomes at six, 12, 24 and 48 months. CARE75+ is a very rich dataset that provides a highly efficient method to investigate how simple instruments might augment eFI performance.
Eligible population
Patients ≥65 years with moderate frailty (eFI score 0.24 to 0.36) or severe frailty (eFI score >0.36) and registered with a ResearchOne, SAIL or LDM practice on 1st April 2018.
All CARE75+ participants with moderate frailty (eFI score 0.24 to 0.36) or severe frailty (eFI score >0.36) will be eligible.
Outcomes for risk prediction (all 12 months)
Predictors
Components of the eFI, supplemented with variables available within routine primary care EHR data and clinical assessment measures practical for use in primary care.
Prognostic models
Each prognostic model will be developed and internally validated in just one of the databases, and then externally validated in a second database
Sample size for prognostic model development
SAIL and ResearchOne extracts will each include ≈600,000 patients aged 65 or over, with an estimated 72,000 having moderate frailty, and 24,000 severe frailty. LDM extract will include ≈150,000 patients aged 65 or over, with an estimated 18,000 having moderate frailty and 6,000 severe frailty.
For model development, a key indicator of the effective sample size is the number of outcome events. Previous research into the outcomes of interest, and feasibility estimates using CARE75+, ResearchOne and SAIL, inform estimates for anticipated number of events within 12 months.
Therefore, even when taking the lowest estimate of incident events by 12 months (for nursing home admission), for each outcome the investigators would expect at least 2,160 events in each of ResearchOne or SAIL. This enables us to robustly estimate a prognostic model for each outcome even with up to 108 predictor parameters, corresponding to 20 events per potential predictor parameter (2160/20). This exceeds 'rule-of-thumb' recommendations of 10 or 15 events per predictor parameter.
Furthermore, conservatively assuming the new models will have a Nagelkerke R-squared of 15%, Riley's sample size formula suggests that at least 7.5 events for each predictor parameter will ensure overfitting and optimism are minimized, when the outcome proportion is 3%. When increasing outcome proportion to 20% (home care package), 9% (fall/fracture), or 15% (mortality), the minimum sample size required is 18, 11.5 and 15 events per predictor parameter, respectively. The investigators exceed all these, due to the large datasets available.
Sample size for external validation
Current recommendations are that at least 100 events and 100 non-events (ideally 200) are required for prognostic model external validation. Our estimates indicate considerably more than this, such as 2160 events for the least prevalent outcome of care home admission in SAIL and ResearchOne, and 540 in LDM (which will only be used for external validation of models).
Missing data
Handled using multiple imputation and Rubin's rules, under a missing at random assumption, including outcome in the imputation model, accounting for practice clustering.
Analysis plan
i) Prognostic modelling
The investigators will build 4 separate prognostic models within the development datasets to predict risk of our 4 key stated outcomes in individuals with moderate or severe frailty as the startpoint.
For each outcome, for those with moderate frailty (eFI score 0.24 to 0.36) or severe frailty (eFI score >0.36) the investigators will develop and internally validate a prognostic model containing just eFI (as a whole as it currently stands) and then containing components of eFI (included as predictors) along with additional routine primary care EHR data. The regression model will be logistic regression or flexible parametric survival, for binary or time-to-event outcomes (as appropriate when the investigators observe the database coding and censoring etc.), to produce outcome risks by 12 months.
Due to the large sample size, overfitting is expected to be small, but the investigators will adjust for it using penalisation via a global shrinkage factor estimated via bootstrapping. Where variable selection is considered important for parsimony, the investigators will rather use penalisation via elastic net. Internal validation will use bootstrapping of the entire development dataset, and optimism-adjusted estimates of predictive performance produced for calibration (e.g. calibration-in-the-large, calibration slope, Observed/Expected), discrimination (e.g. C-statistic) and overall (e.g. Nagelkerke R2) performance of predicted risks. Continuous variables will not be categorised and potential non-linear effects examined using splines or fractional polynomials. Non-proportional hazards for predictors will also be examined with interaction terms with time.
All models will be externally validated in a different database. Predictive performance statistics will be derived as described above (e.g. C-statistic, calibration slope), alongside calibration plots showing agreement between observed and predicted risks, across the spectrum of predicted risks, using a loess non-parameter smoother.
Separately, the investigators will use the CARE75+ dataset to investigate the additional predictive power of clinical assessment measures of prognostic factors identified from the reviews that are practical for use in routine primary care.
ii) Decision modelling
Prognostic models will be translated into a framework to guide clinical decision making by identifying relevant thresholds of predicted risk, above which implementation of our stated interventions is warranted. This will allow us to generate a decision analytic model (DAM), which will be examined using decision curves and net benefit in the external validation datasets.
iii) Health economic evaluation
The health economic evaluation will be conducted in two stages. The objective of the first stage is to provide a short-term, 12-month comparison of the cost-effectiveness of the scenarios identified by the DAM. For the second stage, the investigators will extend our analysis to a long-term cost-effectiveness evaluation of these scenarios.
Inclusion Criteria:
Exclusion Criteria: