Development of a Novel Risk Prediction Tool for Emergency Department Patients Symptoms of Coronary Artery Disease
Development of a Novel Risk Prediction Tool for Emergency Department Patients Symptoms of Coronary Artery Disease
Patients with chest pain and symptoms of acute coronary syndromes (ACS) account for over 600,000 emergency department (ED) visits annually in Canada. 85% of these patients do not have an ACS, and most are discharged from the ED after a thorough evaluation. However, a large proportion of these patients (approximately 180,000 annually) are referred for outpatient objective cardiac testing (exercise stress tests, myocardial perfusion scans, coronary CT angiography) after ED discharge, even though their short-term risk of major adverse cardiac events such as death, new myocardial infarction or need for revascularization is very small. This contributes to substantial low-value healthcare utilization, and limits access for those patients who are likely to benefit from objective testing.
Clinical risk prediction tools can improve the appropriateness of utilization of cardiac testing. However, existing risk prediction tools were developed prior to the advent of new high-sensitivity cardiac troponin assays, were derived in non-representative populations and, when applied to ED patients with low cardiac troponin concentrations, systematically overestimate short-term risk of major adverse cardiac events (MACE).
New, more specific, risk prediction tools are required to accurately guide clinical decision-making for patients who have had an emergency department evaluation for suspected coronary disease. The objective of this research program is to develop individualized risk prediction tools for patients who have had an MI ruled out in the emergency department, to identify patients who are likely to benefit from additional cardiac testing, and to guide the appropriate timing of testing. In other words, the objective is to provide personalized risk estimates to get the Right Patient the Right Test at the Right Time.
We will conduct a multicenter prospective cohort study including emergency department chest pain patients to derive personalized risk prediction tools to distinguish patients at low risk of MACE and not requiring additional cardiac testing from patients who are likely to benefit from additional cardiac testing. We will leverage the existing clinical research infrastructure of the Canadian Emergency Department Rapid Response Network to enrol a large population of representative patients.
The risk prediction tools that we will develop will innovate in the following ways:
5) We will provide granular, individualized risk estimates to guide decision-making around which patients need additional testing.
The knowledge product of this work will improve patient outcomes while also optimizing the appropriate utilization of objective cardiac testing modalities.
Background
Chest pain and symptoms of coronary artery disease are a common cause for presentation to an emergency department <REF>. The key clinical question addressed in the ED is whether these patients have acute myocardial infarction (MI), which is associated with a 30-day mortality risk of 6.1%2. Therefore, patients presenting to the ED with chest pain and symptoms of possible MI typically undergo extensive investigations, including clinical evaluation and testing with electrocardiograms and blood cardiac troponin measurement. Among patients evaluated in the ED for chest pain, up to 10% are diagnosed with MI and admitted to the hospital. Another 5% are admitted for investigations because of concern for unstable angina, a syndrome of chest pain or other symptoms associated with critical coronary stenosis that does manifest a rise in troponin concentration3
The remaining 85% of patients, who have had MI and other high-risk diagnoses ruled out in the ED, have a short-term risk of Major Adverse Cardiac Events (MACE: Death, MI or revascularization) of 2.5% or less1,4,5. Recent guidelines reinforce that patients at low risk of MACE need not undergo additional testing to diagnose coronary disease at the time of their initial hospital encounter. However, in Canada, as many as 40% of these patients are referred for outpatient cardiac testing after ED discharge to screen for undiagnosed coronary disease4,6-8. In the United States, up to 70% of patients at low risk of MACE undergo additional testing, often as inpatients 12-14,35. This an ineffective use of healthcare resources. This low-value testing pattern is driven by reliance on outdated risk scores that overestimate risk of MACE, or use of "gut feeling" estimation of pre-test probability of coronary disease, based on the presence of traditional coronary disease risk factors.
Our objective is to derive, and internally validate, a novel risk prediction tool to accurately predict 30-day MACE among ED patients with chest pain in whom MI has been ruled out and are being considered for ED discharge. This risk prediction tool will guide rational, cost-effective decisions around which patients should be referred for additional cardiovascular testing after a thorough ED evaluation for chest pain. This new risk score will innovate compared to existing risk scores by integrating high-sensitivity troponin testing, specifically accounting for sex and sex-related risk factors, and the presence of pre-existing coronary artery disease.
Methods
We will following broadly accepted methods for risk score development42 to derive, and internally validate, a novel risk score for prediction of 30-day MACE among ED patients with chest pain who have had MI ruled out, using a prospective, multicenter observational cohort study.
Setting
Recruitment will take place in the EDs of large academic hospitals across Canada: Foothills Medical Centre, South Health Campus and Rockyview General Hospital, Calgary; London Health Sciences Centre (University and Victoria Campuses), London; Kingston Health Sciences Centre, Kingston; Sinai Health, Toronto; Sunnybrook Hospital, Toronto; the Ottawa Hospital (Civic and General Campuses), Ottawa; Vancouver General and St. Paul's Hospitals, Vancouver;QEII Health Sciences Centre, Halifax. Each of these EDs has an annual census of over 60,000 patients, including at least 3,000 patients who undergo evaluation for chest pain annually. Consecutive patients will be recruited while research staff are present in the ED, from 8am-8pm, 7days/week at most sites. ED physicians will be asked to collect data independently of research staff during hours that research staff are not present. Eligible patients will receive usual care provided by the ED clinicians. ECG and laboratory testing will be performed as per usual clinical practice.
This study will leverage the data collection and management infrastructure of the Canadian Emergency Research Network (). This is Canada's largest ED research network, formed to systematically collect data on patients tested for SARS-CoV-2 at 50 Canadian EDs. CEDRN's has accrued data on over 203,000 patients tested for SARS-CoV-2 and its data has been used to develop risk scores for a variety of COVID-related outcomes46-48. CEDRN has an established REDCap data management platform and a secure analytic environment.
Financial Support, Study registration and Ethics Review
This project is supported by a Project Grant from the Canadian Institutes of Health Research (CIHR, PJT-191778).
The protocol was approved by the University of Calgary Conjoint Health Research Ethics Board (REB-24-0536) with a waiver of the requirement for informed consent. The study will be approved with a waiver of the requirement for consent at all participating institutions.
Patient population
We will enroll adult patients (age >25) presenting to the ED with chest pain or symptoms consistent with coronary disease, but who do not have an MI or clear alternative diagnosis. Our age cutoff is consistent with prior literature, demonstrating a very low risk of symptomatic coronary disease below age 25.
Inclusion Criteria
Exclusion Criteria
Participant recruitment Eligible patients will be enrolled using a waiver of consent, consistent with Canadian research ethics <REF TCPS2> guidelines for research that poses only minimal risk to participants, to limit selection bias. Data collection variables and outcomes
Symptoms and clinical variables, including pre-test probability of symptomatic coronary disease will be recorded by ED physicians. Data collection from physicians will employ several approaches, depending on site-specific workflows. Sites will use a combination of: 1) Direct data entry into a REDCap data collection tool linked form the site's electronic medical record or from QR codes posted in the ED; 2) Standardized clinical documentation templates that include all physcian-recorded variables; 3) Paper data collection forms to be transcribed into REDCap by site research staff; 4) Staff- or patient-completed data collection forms verified by physicans, with physician completion of specific fields. Demographics, ECG, and laboratory results will be recorded on an ED data collection form completed at the index ED visit by local research staff. Demographics of any missed eligible patients will be quantified to verify the absence of selection bias.
Outcomes
The primary outcome for the prediction tool will be the incidence of MACE within 30 days after the index ED visit. MACE is defined as all-cause mortality, MI, or revascularization (non-elective coronary bypass grafting or percutaneous coronary intervention). Secondary outcomes include individual MACE components: all-cause mortality, MI, and revascularization within 90 days of the index ED visit.
Outcomes will be ascertained first by querying health records and administrative databases of participating hospitals. This approach has been robust in prior studies conducted by this team43,50-55 and will identify most outcomes as the participating sites are the coronary revascularization centers for their geographic region. Outcomes occurring outside of the participating sites will be ascertained by linking to provincial vital statistics, Discharge Abstract Database (DAD) and National Ambulatory Care Reporting System (NACRS) databases via the Canadian Institute of Health Information)Cardiovascular outcome identification algorithms using these administrative data sources have been previously validated and used in prior studies7,8,56. All outcomes will be centrally adjudicated by a committee comprised of cardiologists and emergency physicians.
Data flow and management
CEDRN's REDCap-based data management platform, housed on a secure research environment at PopDataBC, will be used for data management and analysis. All data will be either directly enterd by physicians or entered by participating sites into an electronic case report form developed using the REDCap electronic data capture platform.
After data quality assurance has been completed within the REDCap database, the data will be transferred to a secure research environment housed at PopDataBC. This secure research environment will accessible only to study analysts with an electronic access key and will be used as the analytic platform.
No participant identifiers will be transferred to the REDCap database. Each site will retain a master study list containing the participant's unique study ID and provincial health number for linkage.
Sites will transfer this master list to the CEDRN Provincial coordinating site (Vancouver General Hospital/UBC, Foothills Medical Centre/University of Calgary, Kingston General Hospital/KGH). This will then be sent securely to CIHI for linkage to administrative datasets. CIHI will then securely transfer de-identified outcome data (containing only the participant's unique study ID) to the PopDataBC secure research environment.
3.6 Analysis: Prediction rule derivation and internal validation
Descriptive statistics (means, standard deviation, frequency, and/or percentages) will be used to summarize patients' the demographic and clinical characteristics. Patients' demographic and clinical characteristics will be used to predict patients' probability of having 30-day MACE.
For continuous predictors, restricted cubic splines and loess functions will be used to assess the most appropriate functional form of the relationship between each continuous variable and the MACE outcome. Hs-cTn will be modelled in different ways (e.g., absolute value, multiples of limit of quantification, proportion of 99th percentile rather than absolute concentration, absolute/relative change from baseline). Team members have previously developed analytic approaches that can combine different hs-cTn assays to develop a prognostic tool57. The association of each functional form of troponin variable with the outcome quantified, and the troponin variable having the strongest association with the outcome will be included in the multivariable model. To prevent overfitting due to a large number of candidate variables, and to develop a user-friendly risk score that can easily be used at the bedside, some variables may be combined into composite predictors. For example, rather than including all coronary risk factors in a model, we will create a composite categorical variable indicating the presence of three or more coronary risk factors as opposed to including each individual risk factor in the model. Rather than including all granular symptom variables, we may create a composite "classical anginal symptoms" variable. These will be created with expert clinician input to ensure sensibility and utility to end-users.
The MACE risk prediction model will be developed using a logistic regression model that includes patients' demographic and clinical characteristics as candidate predictors. Current evidence suggests that regression models have comparable predictive accuracy to machine learning algorithms for risk prediction58,59. To fit the most parsimonious set of predictors which maximizes the model's predictive accuracy the LASSO method will be used. LASSO is a penalized estimation technique which simultaneously achieves variable selection and coefficient shrinkage to mitigate overfitting60. Through this selection and shrinkage process LASSO can also effectively handle collinearity. Interactions between sex and other predictor variables will be quantified and, if indicated, sex-specific models will be created.
The discriminatory performance of the model will be evaluated using sensitivity, specificity, the area under the receiver operating characteristic curve (AUC), and F1 score. To account for the imbalance in the dataset, the classification of each individual based on logistic regression will be weighted using the inverse of the ratio of the size of the "Event" to the "No Event" group as the penalty for misclassification each individual. For example, the cost of misclassifying a patient at high risk of MACE in 30 days will be higher than the cost of misclassifying an individual with small risk of MACE. The calibration performance of the model will be derived using Brier score and graphically. In the latter, we will graphically compare predicted and observed proportion of the outcome across deciles of model predicted probabilities.
To internally validate the 30-day MACE risk model, repeated 10-fold cross validation will be used. This will be done by randomly splitting the data into 9:1 ratio, retraining the model in the nine-tenth of the data and estimating the accuracy in the one-tenth. This process will be repeated 10 times. The final accuracy of the model is obtained as the average discrimination and calibration measures (AUC, sensitivity, specificity, F1 score, Brier score). We will externally validate the trained model in a temporally and geographically distinct cohort of eligible patients in a future study.
An integer-based risk score will be created from the prediction model by generating regression coefficient-based point scores61,62. This will simplify the estimation of patient's risk by assigning integer values to each level of each predictor, allowing clinicians to easily estimate risk by summing integers. The discrimination and calibration of the integer-based score will be evaluated as above, and sensitivity and specificity at different score cut-offs will be quantified.
We will compare the performance of the novel risk score (sensitivity, specificity, proportion of patients classified as low risk) to existing risk scores using net reclassification indices and decision curve analyses. We will compare observed physician referral patterns to expected referral patterns based on the integer-based risk score. We will also conduct sensitivity analyses stratifying sites by outpatient testing rates to identify any differences in outcome rates attributable to testing patterns and will conduct secondary analyses to validate the risk score in samples from high- vs. low-referral rate sites. All analyses will be conducted using Python software (Version 2.7, available at www.python.org) and R (R-project.org). PopDataBC integrates RStudio for analysis within its secure research environment.
3.7 Sample Size and precision
Current risk scores, when used in conjunction with hs-cTn testing, have a specificity no better than 60%29. Our goal is to develop a score with a specificity of 70%. This analysis is designed to maximize specificity while maintaining a fixed sensitivity. We propose a target sensitivity of 98.5% for a dichotomized score, as this is the standard to which other risk scores have been developed24,26 and approximates emergency physicians' documented risk tolerance of 1-2% for missed cardiac events64. Data from an 1184 patient study from our lead site50,51 suggests that the 30-day MACE risk among patients with MI ruled out using high-sensitivity troponin is approximately 2.5%. We will conservatively estimate the 30-day MACE risk in our cohort at 2% for the purposes of sample size calculation, which will require a large cohort to ensure acceptable precision around our target sensitivity. A sample of 6350 patients, with 127 events, would give an acceptable precision around a target sensitivity of 98.5% for a dichotomized risk score. The study will be overpowered for specificity, with excellent precision around a range of specificity point estimates. Appendix 3 details precision around various combinations of sensitivity and specificity with different permutations of sample size and 30-day MACE rate. Each participating site has experience recruiting patients with chest pain in previous observational studies1,4,41,50-52,63,65,66. Each site was able to recruit approximately 600 patients per year in previous studies. To achieve a total sample size of 6350 patients, the ten sites will conservatively require 18 months of recruitment.
Inclusion Criteria:
Exclusion Criteria:
amcrae@ucalgary.ca403-210-8835
ajmacci@ucalgary.ca