Not provided
| ID | Type | Description | Link |
|---|---|---|---|
| 5P30AG034532-12 | U.S. NIH Grant/Contract | View source |
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Name | Class |
|---|---|
| National Institute on Aging (NIA) | NIH |
Not provided
Not provided
Not provided
Not provided
The purpose of the current study is to test different interventions to determine the most effective way to promote flu vaccine uptake in a high-risk population identified by an "artificial intelligence" (AI) or machine learning (ML) algorithm. The specific aims are:
Background
On average, 8% of the US population gets sick from flu each flu season (Tokars et al. 2018). Since 2010, the annual disease burden of influenza has included 9-45 million illnesses, 140,000-810,000 hospitalizations, and 12,000-61,000 deaths (CDC 2020). The CDC recommends the flu vaccination to everyone aged 6+ months, with rare exception; almost anyone can benefit from the vaccine, which can reduce illnesses, missed work, hospitalizations, and death (CDC 2019a). Flu vaccination will be especially important for high-risk patients during the COVID-19 pandemic so that flu cases are reduced and resources conserved.
While most recover from influenza without treatment, the elderly, those with comorbidities, and other high-risk individuals can experience complications such as pneumonia, other respiratory illness, and death. Geisinger, a large health system in Pennsylvania and New Jersey, has partnered with Medial EarlySign (Medial; www.earlysign.com) to develop a machine learning (ML) algorithm to identify patients at risk for serious (moderate to severe) flu-associated complications on the basis of their existing electronic health record (EHR) data. Geisinger will deploy this system during the 2020-21 flu season and contact the identified patients with special messages (in addition to standard efforts made by the health system every flu season) to encourage vaccination. Flu vaccination will be especially important for high-risk patients during the COVID-19 pandemic so that flu cases are reduced and resources conserved.
Published results suggest Medial's ML systems identify high-risk patients in other contexts (Goshen et al., 2018; Zack et al., 2019). However, there is little evidence about (a) whether informing patients they are at high risk makes them more likely to receive vaccination; (b) how patients react to being told their risk status is the result of an analysis of their health records; and (c) whether informing patients their risk status has been determined by an "algorithm," by "machine learning," and/or by "artificial intelligence" will increase or decrease their likelihood of getting vaccinated. This study will address these gaps in the literature, which are especially important in light of the anticipated future growth of AI/ML system use throughout healthcare.
Medial's algorithm is an example of how interoperable health information exchange (HIE)-the ability for health information technology to share patient data-can improve the efficiency and effectiveness of healthcare. However, patients may not appreciate these benefits or the fact that healthcare has become substantially more integrated and collaborative. A systematic review of patient privacy concerns about HIE found that 15-74% of patients expressed privacy concerns, depending on the study, and concluded that patient perspectives remain poorly understood. A flu outreach message that explicitly references a review of patient medical records might backfire as patients react badly to a sense they have lost control of their health records.
There is conflicting evidence on how people respond to advice or information that comes from an algorithm or machine. Dietvorst et al. (2015) documented a pattern of "algorithm aversion," in which people choose inferior human over superior algorithmic forecasts, especially after they observed the algorithm make an error. In contrast, Logg et al. (2018) described "algorithm appreciation," in which people followed advice more when they thought it came from algorithms than when they thought it came from human beings. Finally, Bigman and Gray (2019) found aversion to algorithms that make "moral decisions," including a (fictitious) medical decision of choosing whether or not to operate on a high-risk patient. In the current setting, the algorithm is merely advising patients on taking an action (an annual flu shot) that is already the standard of care, and there is no opportunity to observe an erroneous recommendation, so the hypothesis is that "algorithm appreciation" will cause people to react positively to being informed of the algorithm's role. Thus, this study will address two important research questions:
Our specific aims are:
Research Strategy
Included in the study will be current Geisinger patients 17+ years of age with one or more visits to a Geisinger primary care physician (PCP) between January 1, 2008 and January 30, 2020 and no contraindications for flu vaccine. Medial will provide flu-complication risk scores from their ML algorithm (based on coded EHR data), on the basis of which the top 10% of patients at highest risk will be included. Based on prior behavior and other predictors in a second ML model, Medial will also provide the likelihood each patient will get vaccinated during the study flu season; these values and the primary risk scores will be used as covariates in exploratory data analyses. The anticipated number of patients in the top 10% of risk is 56,000.
On average in the last 3 flu seasons, 55% of Geisinger patients aged 65+ are vaccinated each season, so we will use this as a proxy base rate for a control condition in our power analysis. The study will have 92% power to detect a 2% absolute difference or greater in the vaccination outcome between conditions (55% vs 57%, two-tailed alpha of .05), on the assumption that each condition will have 56,000/4=14,000 patients. For the rarer outcome of flu diagnosis, we have 95% power to detect a 0.8% absolute difference or greater-from an estimated 3.9% rate in this high-risk population (based on the CDC estimate for people age 65+ [Tokars et al., 2018]) to a 3.1% rate.
The primary study outcomes will be the rates of flu vaccination and flu diagnoses during the 2020-21 season (September-March) by targeted patients. Secondary, exploratory outcomes will also be measured: Rates of flu vaccination and diagnoses by fellow household members of targeted patients; rates of flu vaccination and diagnoses by non-targeted patients who were assigned a risk score that fell just below the cutoff of targeted patients ("sub-threshold risk"); rates of flu complications and flu-like symptoms among targeted patients, household members, and those at sub-threshold risk; and rates of other relevant healthcare utilization outcomes such as ER visits and hospitalizations.
Generalized linear mixed models (GLMMs) will examine the primary study outcomes as a function of the study arms (between-subjects), with patient-visited PCPs and/or clinics included as random effects variables, assuming high intraclass correlation coefficients. GLMMs will specify a binary distribution and log-link function in the case of dichotomous outcome variables (e.g., flu vaccination, flu diagnosis), and a negative binomial distribution and log-link function in the case of any highly positively skewed count variables such as ER visits and hospitalizations (where over-dispersion typically remains in the case of a Poisson distribution model). For these exploratory analyses, within-patient change (from the same period one year earlier) will also be analyzed. Also, each patient will receive the same type of communication (a/b/c/d) via up to three modalities-printed letter to their mailing address, SMS to their mobile phone, and/or secure message via Geisinger's patient portal-depending on what information is on file for each patient. The communication channels used for each patient will be covariates in later analyses.
*Note: The study will not necessarily use the terms "AI," "ML," or "algorithm" in the messages to groups b, c, and d; instead, these messages will be designed to be readable and comprehensible by the patient audience while still including the key concepts that differentiate the interventions from one another.
Not provided
Not provided
Not provided
Not provided
| Label | Type | Description | Intervention Names |
|---|---|---|---|
| Control | No Intervention | This group receives no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted. | |
| High risk only | Experimental | This group receives messages telling them they have been identified to be at high risk for flu complications without specifying how or why the health system believes this to be the case. |
|
| High risk based on medical records | Experimental | This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records. |
|
| High risk based on algorithm | Experimental | This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by an AI/ML system. |
|
| Sub-threshold patients |
| Name | Type | Description | Arm Group Labels | Other Names |
|---|---|---|---|---|
| Risk reduction | Behavioral | Mailed letter, SMS, and/or patient portal message |
|
| Measure | Description | Time Frame |
|---|---|---|
| Flu Vaccination Rate | Patient received a flu vaccination | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
| Flu Vaccination Rate by Risk Level | Patient received a flu vaccination Note: For patients who received risk communications, those in the top 3% were always told they were in the top 3% of risk. Those in the top 4-10% of risk were randomized to be told that they were in the top 10% of risk or high risk. Control patients in the top 3% and top 4-10% of risk were allocated to the top 3% and randomized to either top 10% or high risk groups, respectively, at the same time as those in the patient contact groups, even though these control patients were not contacted. | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
| High Confidence Flu Diagnosis Rate | Patient received a flu diagnosis via a positive PCR/antigen/molecular test | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
| Measure | Description | Time Frame |
|---|---|---|
| "Likely Flu" Diagnosis Rate | Patient received a diagnosis that was likely flu, as assessed via ICD codes or Tamiflu administration or positive PCR/antigen/molecular test. Note that this outcome is a superset of the "high confidence flu diagnosis rate" outcome. | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
Not provided
Inclusion Criteria:
Exclusion Criteria:
Not provided
Not provided
Not provided
Not provided
Not provided
Not provided
| Name | Affiliation | Role |
|---|---|---|
| Christopher Chabris, PhD | Geisinger Clinic | Principal Investigator |
| Facility | Status | City | State | ZIP | Country | Contacts |
|---|---|---|---|---|---|---|
| Geisinger | Danville | Pennsylvania | 17822 | United States |
| PubMed Identifier | Type | Citation | Retractions |
|---|---|---|---|
| 30107256 | Background | Bigman YE, Gray K. People are averse to machines making moral decisions. Cognition. 2018 Dec;181:21-34. doi: 10.1016/j.cognition.2018.08.003. Epub 2018 Aug 11. | |
| 25401381 | Background | Dietvorst BJ, Simmons JP, Massey C. Algorithm aversion: people erroneously avoid algorithms after seeing them err. J Exp Psychol Gen. 2015 Feb;144(1):114-26. doi: 10.1037/xge0000033. Epub 2014 Nov 17. |
Not provided
Not provided
Data with no personally identifiable information will be made available to other researchers on the Open Science Framework for transparency. This will include the essential data and code needed to replicate the analysis that yielded reported findings.
The data will become available after publication of study results in a scientific journal and will be available as long as the Open Science Framework hosts the data.
The data and code are available in the OSF repository at the url below in the folder ClinicalTrials.gov data/Study 1.
Not provided
Not provided
Not provided
| ID | Title | Description |
|---|---|---|
| FG000 | Control | This group receives no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted. |
| FG001 | High Risk Only | This group receives messages telling them they have been identified to be at high risk for flu complications without specifying how or why the health system believes this to be the case. Risk reduction: Mailed letter, SMS, and/or patient portal message |
| FG002 | High Risk Based on Medical Records | This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records. Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message |
| FG003 | High Risk Based on Algorithm | This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by an AI/ML system. Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message |
| FG004 | Sub-threshold Patients | Patients in this group are in the top 11-20% of risk for flu and complications, slightly lower risk than those included in the intervention, who are in the top 10% of risk for flu and complications. This group of patients does not receive an intervention, but is monitored for flu shots as a comparison to target patients. |
| FG005 | Household Members | This group of patients share an address with target high-risk patients (in arms 1-4). This group does not receive an intervention but is monitored for spillover effects of the intervention. Note that some household members of target patients were also sub-threshold risk. Additionally, some of these patients were household members of more than one target patient. The numbers reported here reflect unique household members who were not also sub-threshold risk patients. |
| Title | Milestones | Reasons Not Completed | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall Study |
|
This is the number of participants who were randomized to each arm. Some randomized participants were excluded from analysis if they were vaccinated prior to the study start date or if they were contraindicated for flu vaccine.
Not provided
| ID | Title | Description |
|---|---|---|
| BG000 | Control | This group receives no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted. |
| Units | Counts |
|---|---|
| Participants |
|
| Title | Description | Population Description | Parameter Type | Dispersion Type | Unit of Measure | Calculate Percentage | Denominator Units Selected | Denominators | Classes |
|---|---|---|---|---|---|---|---|---|---|
| Age, Continuous | Mean |
| Type | Title | Description | Population Description | Reporting Status | Anticipated Posting Date | Parameter Type | Dispersion Type | Unit of Measure | Calculate Percentage | Time Frame | Units Analyzed | Denominator Units Selected | Arm/Group Information | Denominators | Classes | Analyses | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Primary | Flu Vaccination Rate | Patient received a flu vaccination | Participants were excluded from analysis if they were vaccinated prior to the study start date or if they were contraindicated for flu vaccine. | Posted | Count of Participants | Participants | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
Adverse events were not monitored. See description.
This is a minimal risk study, and has received expedited IRB review in keeping with 45 CFR §46.110. EHR and claims data will report inputs (i.e., the subject's assigned arm) and outcomes (i.e., flu vaccination, flu diagnosis, flu-like symptoms); we will therefore be unable to monitor this data for AE/SAEs other than the study outcomes themselves.
Not provided
| ID | Title | Description | Deaths (Affected) | Deaths (At Risk) | Serious Events (Affected) | Serious Events (At Risk) | Other Events (Affected) | Other Events (At Risk) |
|---|---|---|---|---|---|---|---|---|
| EG000 | Control | This group receives no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted. |
Not provided
Not provided
One primary and multiple secondary outcomes involved analysis of flu diagnoses and complications. However, flu cases and other related outcomes in the 2020-2021 flu season were too low to detect any difference in these more distal outcomes, so results are limited to flu vaccination outcomes.
| Title | Organization | Phone | Extension | |
|---|---|---|---|---|
| Amir Goren, PhD | Geisinger Health | 570-214-4395 | agoren@geisinger.edu |
Not provided
| Type | Includes Protocol | Includes SAP | Includes ICF | Document Label | Document Date | Document Uploaded Date | Document File Name |
|---|---|---|---|---|---|---|---|
| Prot | Yes | No | No | Study Protocol | Sep 13, 2021 | Nov 12, 2021 | Prot_001.pdf |
| SAP | No | Yes | No | Statistical Analysis Plan | Jan 8, 2021 | Jan 8, 2021 | SAP_000.pdf |
Not provided
| ID | Term |
|---|---|
| D007251 | Influenza, Human |
| D015438 | Health Behavior |
| D040242 | Risk Reduction Behavior |
| ID | Term |
|---|---|
| D012141 | Respiratory Tract Infections |
| D007239 | Infections |
| D009976 | Orthomyxoviridae Infections |
| D012327 | RNA Virus Infections |
Not provided
Not provided
| ID | Term |
|---|---|
| D061366 | Numbers Needed To Treat |
| ID | Term |
|---|---|
| D018401 | Sample Size |
| D012107 | Research Design |
| D008722 | Methods |
| D008919 | Investigative Techniques |
Not provided
Not provided
Patients from the high-risk sample will be randomly assigned to one of 4 groups:
Control: group that receives no additional pro-vaccination intervention beyond Geisinger's normal efforts.
High Risk Only: group that receives messages telling them they have been identified to be at high risk for flu complications without specifying how/why Geisinger believes this to be the case
High Risk Based on Medical Records: group that receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records
High Risk Based on Algorithm: group that receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by AI/ML
Two additional groups will be monitored for outcome data:
Sub-threshold patients: patients who are in the top 11-20% of risk
Household members: patients who share an address with target patients
Not provided
Not provided
Participants (i.e., patients) will not be informed specifically of their assignment to different arms throughout the study. Providers who prescribe vaccination and diagnose conditions will not be randomized to study arms or informed of patient assignment.
| No Intervention |
Patients in this group is in the top 11-20% of risk for flu and complications, slightly lower risk than those included in the intervention, who are in the top 10% of risk for flu and complications. This group of patients does not receive an intervention, but are monitored for flu shots as a comparison to target patients. |
| Household members | No Intervention | This group of patients share an address with target high-risk patients (in arms 1-4). This group does not receive an intervention but is monitored for spillover effects of the intervention. |
| Medical records-based recommendation | Behavioral | Mailed letter, SMS, and/or patient portal message |
|
|
| Algorithm-based recommendation | Behavioral | Mailed letter, SMS, and/or patient portal message |
|
|
| Flu Complications Rate | Patient was diagnosed with flu-related complications | Through 3 months after the end of the flu season (August 31st 2021), approximately 12 months assessment duration |
| Change in ER Visits From Pre- to Post-intervention | Number of patient visits to the ER, examining relative rate of visits across Time 1 and 2 | Within 12 months pre-intervention (Time 1) and within 12 months post-intervention (Time 2) |
| Change in Hospitalizations From Pre- to Post-intervention | Number of patient hospital visits, examining relative rate of visits across Time 1 and 2 | Within 12 months pre-intervention (Time 1) and within 12 months post-intervention (Time 2) |
| Flu Vaccination Among Fellow Household Members | Non-targeted fellow household members of targeted patients received a flu vaccination | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
| High Confidence Flu Diagnosis Among Fellow Household Members | Non-targeted fellow household members of targeted patients received a flu diagnosis (via a positive PCR/antigen/molecular test) | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
| "Likely Flu" Diagnosis Among Fellow Household Members | Non-targeted fellow household members of targeted patients received a diagnosis that was likely flu (as assessed via ICD codes or Tamiflu administration or positive PCR/antigen/molecular test) | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
| Flu Complications Among Fellow Household Members | Non-targeted fellow household members of targeted patients were diagnosed with flu-related complications | Through 3 months after the end of the flu season (August 31st 2021), approximately 12 months assessment duration |
| Flu Vaccination Among Those at Sub-threshold Risk | Non-targeted sub-threshold risk patients received a flu vaccination | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
| High Confidence Flu Diagnosis Among Those at Sub-threshold Risk | Non-targeted sub-threshold risk patients received a flu diagnosis (via a positive PCR/antigen/molecular test) | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
| "Likely Flu" Diagnosis Among Those at Sub-threshold Risk | Non-targeted sub-threshold risk patients received a diagnosis that was likely flu (as assessed via ICD codes or Tamiflu administration or positive PCR/antigen/molecular test) | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
| Flu Complications Among Those at Sub-threshold Risk | Non-targeted sub-threshold risk patients were diagnosed with flu-related complications | Through 3 months after the end of the flu season (August 31st 2021), approximately 12 months assessment duration |
| 30652563 | Background | Goshen R, Choman E, Ran A, Muller E, Kariv R, Chodick G, Ash N, Narod S, Shalev V. Computer-Assisted Flagging of Individuals at High Risk of Colorectal Cancer in a Large Health Maintenance Organization Using the ColonFlag Test. JCO Clin Cancer Inform. 2018 Dec;2:1-8. doi: 10.1200/CCI.17.00130. |
| Background | Logg, J.M., Minson, J.A., & Moore, D.A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90-103. https://doi.org/10.1016/j.obhdp.2018.12.005 |
| 30914173 | Background | Shen N, Bernier T, Sequeira L, Strauss J, Silver MP, Carter-Langford A, Wiljer D. Understanding the patient privacy perspective on health information exchange: A systematic review. Int J Med Inform. 2019 May;125:1-12. doi: 10.1016/j.ijmedinf.2019.01.014. Epub 2019 Feb 1. |
| 19935413 | Background | Rothberg MB, Haessler SD. Complications of seasonal and pandemic influenza. Crit Care Med. 2010 Apr;38(4 Suppl):e91-7. doi: 10.1097/CCM.0b013e3181c92eeb. |
| 29206909 | Background | Tokars JI, Olsen SJ, Reed C. Seasonal Incidence of Symptomatic Influenza in the United States. Clin Infect Dis. 2018 May 2;66(10):1511-1518. doi: 10.1093/cid/cix1060. |
| 14609480 | Background | Turner D, Wailoo A, Nicholson K, Cooper N, Sutton A, Abrams K. Systematic review and economic decision modelling for the prevention and treatment of influenza A and B. Health Technol Assess. 2003;7(35):iii-iv, xi-xiii, 1-170. doi: 10.3310/hta7350. |
| 23741777 | Background | WHO Guidelines for Pharmacological Management of Pandemic Influenza A(H1N1) 2009 and Other Influenza Viruses. Geneva: World Health Organization; 2010 Feb. Available from http://www.ncbi.nlm.nih.gov/books/NBK138515/ |
| 31255564 | Background | Zack CJ, Senecal C, Kinar Y, Metzger Y, Bar-Sinai Y, Widmer RJ, Lennon R, Singh M, Bell MR, Lerman A, Gulati R. Leveraging Machine Learning Techniques to Forecast Patient Prognosis After Percutaneous Coronary Intervention. JACC Cardiovasc Interv. 2019 Jul 22;12(14):1304-1311. doi: 10.1016/j.jcin.2019.02.035. Epub 2019 Jun 26. |
| Background | Centers for Disease Control and Prevention. (2020). Disease Burden of Influenza. https://www.cdc.gov/flu/ about/burden/index.html (Jan 10). |
| Background | Centers for Disease Control and Prevention. (2019a). Who Needs a Flu Vaccine and When. https://www.cdc.gov/flu/prevent/vaccinations.htm (Oct 11). |
| Background | Centers for Disease Control and Prevention. (2019b). Flu Vaccination Coverage, United States, 2018-19 Influenza Season. https://www.cdc .gov/flu/fluvaxview/coverage-1819estimates.htm |
| BG001 |
| High Risk Only |
This group receives messages telling them they have been identified to be at high risk for flu complications without specifying how or why the health system believes this to be the case. Risk reduction: Mailed letter, SMS, and/or patient portal message |
| BG002 | High Risk Based on Medical Records | This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records. Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message |
| BG003 | High Risk Based on Algorithm | This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by an AI/ML system. Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message |
| BG004 | Sub-threshold Patients | Patients in this group are in the top 11-20% of risk for flu and complications, slightly lower risk than those included in the intervention, who are in the top 10% of risk for flu and complications. This group of patients does not receive an intervention, but is monitored for flu shots as a comparison to target patients. |
| BG005 | Household Members | This group of patients share an address with target high-risk patients (in arms 1-4). This group does not receive an intervention but is monitored for spillover effects of the intervention. Note that some household members of target patients were also sub-threshold risk. Additionally, some of these patients were household members of more than one target patient. The numbers reported here reflect unique household members who were not also sub-threshold risk patients. |
| BG006 | Total | Total of all reporting groups |
| years |
|
| Sex: Female, Male | Count of Participants | Participants |
|
| Ethnicity (NIH/OMB) | Count of Participants | Participants |
|
| Race/Ethnicity, Customized | Count of Participants | Participants |
|
| Region of Enrollment | Number | participants |
|
This group receives messages telling them they have been identified to be at high risk for flu complications without specifying how or why the health system believes this to be the case. Risk reduction: Mailed letter, SMS, and/or patient portal message |
| OG002 | High Risk Based on Medical Records | This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records. Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message |
| OG003 | High Risk Based on Algorithm | This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by an AI/ML system. Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message |
|
|
|
| Primary | Flu Vaccination Rate by Risk Level | Patient received a flu vaccination Note: For patients who received risk communications, those in the top 3% were always told they were in the top 3% of risk. Those in the top 4-10% of risk were randomized to be told that they were in the top 10% of risk or high risk. Control patients in the top 3% and top 4-10% of risk were allocated to the top 3% and randomized to either top 10% or high risk groups, respectively, at the same time as those in the patient contact groups, even though these control patients were not contacted. | Participants were excluded from analysis if they were vaccinated prior to the study start date or if they were contraindicated for flu vaccine. | Posted | Count of Participants | Participants | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
|
|
|
| Primary | High Confidence Flu Diagnosis Rate | Patient received a flu diagnosis via a positive PCR/antigen/molecular test | Flu cases and other related outcomes (flu complications, ER visits for flu, hospitalizations for flu) in the 2020-2021 flu season were too low to detect any meaningful differences across study arms. Therefore, the study team did not collect data for this outcome. | Posted | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
|
| Secondary | "Likely Flu" Diagnosis Rate | Patient received a diagnosis that was likely flu, as assessed via ICD codes or Tamiflu administration or positive PCR/antigen/molecular test. Note that this outcome is a superset of the "high confidence flu diagnosis rate" outcome. | Flu cases and other related outcomes (flu complications, ER visits for flu, hospitalizations for flu) in the 2020-2021 flu season were too low to detect any meaningful differences across study arms. Therefore, the study team did not collect data for this outcome. | Posted | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
|
| Secondary | Flu Complications Rate | Patient was diagnosed with flu-related complications | Flu cases and other related outcomes (flu complications, ER visits for flu, hospitalizations for flu) in the 2020-2021 flu season were too low to detect any meaningful differences across study arms. Therefore, the study team did not collect data for this outcome. | Posted | Through 3 months after the end of the flu season (August 31st 2021), approximately 12 months assessment duration |
|
|
| Secondary | Change in ER Visits From Pre- to Post-intervention | Number of patient visits to the ER, examining relative rate of visits across Time 1 and 2 | Flu cases and other related outcomes (flu complications, ER visits for flu, hospitalizations for flu) in the 2020-2021 flu season were too low to detect any meaningful differences across study arms. Therefore, the study team did not collect data for this outcome. | Posted | Within 12 months pre-intervention (Time 1) and within 12 months post-intervention (Time 2) |
|
|
| Secondary | Change in Hospitalizations From Pre- to Post-intervention | Number of patient hospital visits, examining relative rate of visits across Time 1 and 2 | Flu cases and other related outcomes (flu complications, ER visits for flu, hospitalizations for flu) in the 2020-2021 flu season were too low to detect any meaningful differences across study arms. Therefore, the study team did not collect data for this outcome. | Posted | Within 12 months pre-intervention (Time 1) and within 12 months post-intervention (Time 2) |
|
|
| Secondary | Flu Vaccination Among Fellow Household Members | Non-targeted fellow household members of targeted patients received a flu vaccination | Household members were excluded from analysis if they were vaccinated prior to the study start date or if they were contraindicated for flu vaccine. Household members who shared an address with multiple patients in the intervention were counted in this outcome once for every eligible target participant who shared their address. | Posted | Count of Participants | Participants | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
|
|
| Secondary | High Confidence Flu Diagnosis Among Fellow Household Members | Non-targeted fellow household members of targeted patients received a flu diagnosis (via a positive PCR/antigen/molecular test) | Flu cases and other related outcomes (flu complications, ER visits for flu, hospitalizations for flu) in the 2020-2021 flu season were too low to detect any meaningful differences across study arms. Therefore, the study team did not collect data for this outcome. | Posted | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
|
| Secondary | "Likely Flu" Diagnosis Among Fellow Household Members | Non-targeted fellow household members of targeted patients received a diagnosis that was likely flu (as assessed via ICD codes or Tamiflu administration or positive PCR/antigen/molecular test) | Flu cases and other related outcomes (flu complications, ER visits for flu, hospitalizations for flu) in the 2020-2021 flu season were too low to detect any meaningful differences across study arms. Therefore, the study team did not collect data for this outcome. | Posted | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
|
| Secondary | Flu Complications Among Fellow Household Members | Non-targeted fellow household members of targeted patients were diagnosed with flu-related complications | Flu cases and other related outcomes (flu complications, ER visits for flu, hospitalizations for flu) in the 2020-2021 flu season were too low to detect any meaningful differences across study arms. Therefore, the study team did not collect data for this outcome. | Posted | Through 3 months after the end of the flu season (August 31st 2021), approximately 12 months assessment duration |
|
|
| Secondary | Flu Vaccination Among Those at Sub-threshold Risk | Non-targeted sub-threshold risk patients received a flu vaccination | Participants were excluded from analysis if they were vaccinated prior to the study start date or if they were contraindicated for flu vaccine. | Posted | Count of Participants | Participants | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
|
|
| Secondary | High Confidence Flu Diagnosis Among Those at Sub-threshold Risk | Non-targeted sub-threshold risk patients received a flu diagnosis (via a positive PCR/antigen/molecular test) | Flu cases and other related outcomes (flu complications, ER visits for flu, hospitalizations for flu) in the 2020-2021 flu season were too low to detect any meaningful differences across study arms. Therefore, the study team did not collect data for this outcome. | Posted | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
|
| Secondary | "Likely Flu" Diagnosis Among Those at Sub-threshold Risk | Non-targeted sub-threshold risk patients received a diagnosis that was likely flu (as assessed via ICD codes or Tamiflu administration or positive PCR/antigen/molecular test) | Flu cases and other related outcomes (flu complications, ER visits for flu, hospitalizations for flu) in the 2020-2021 flu season were too low to detect any meaningful differences across study arms. Therefore, the study team did not collect data for this outcome. | Posted | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
|
| Secondary | Flu Complications Among Those at Sub-threshold Risk | Non-targeted sub-threshold risk patients were diagnosed with flu-related complications | Flu cases and other related outcomes (flu complications, ER visits for flu, hospitalizations for flu) in the 2020-2021 flu season were too low to detect any meaningful differences across study arms. Therefore, the study team did not collect data for this outcome. | Posted | Through 3 months after the end of the flu season (August 31st 2021), approximately 12 months assessment duration |
|
|
| Post-Hoc | Flu Vaccination Rate by Race | Patient received a flu vaccination | Participants were excluded from analysis if they were vaccinated prior to the study start date or if they were contraindicated for flu vaccine. | Posted | Count of Participants | Participants | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
|
|
| Post-Hoc | Flu Vaccination Rate by Gender | Patient received a flu vaccination | Participants were excluded from analysis if they were vaccinated prior to the study start date or if they were contraindicated for flu vaccine. | Posted | Count of Participants | Participants | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
|
|
| Post-Hoc | Flu Vaccination Rate by Ethnicity | Patient received a flu vaccination | Participants were excluded from analysis if they were vaccinated prior to the study start date or if they were contraindicated for flu vaccine. | Posted | Count of Participants | Participants | Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration |
|
|
|
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| EG001 | High Risk Only | This group receives messages telling them they have been identified to be at high risk for flu complications without specifying how or why the health system believes this to be the case. Risk reduction: Mailed letter, SMS, and/or patient portal message | 0 | 0 | 0 | 0 | 0 | 0 |
| EG002 | High Risk Based on Medical Records | This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records. Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message | 0 | 0 | 0 | 0 | 0 | 0 |
| EG003 | High Risk Based on Algorithm | This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by an AI/ML system. Risk reduction: Mailed letter, SMS, and/or patient portal message Medical records-based recommendation: Mailed letter, SMS, and/or patient portal message Algorithm-based recommendation: Mailed letter, SMS, and/or patient portal message | 0 | 0 | 0 | 0 | 0 | 0 |
Not provided
Not provided
| D014777 | Virus Diseases |
| D012140 | Respiratory Tract Diseases |
| D001519 | Behavior |
|
| Top 10% |
|
|
| High risk |
|
|
| This analysis tested whether vaccination differed in patients with the same numeric risk phrasing are differentially affected due to different specific risk levels (3% vs. 10%). This analysis was limited to those informed they were high risk (High risk only, High risk based on medical records, High risk based on algorithm). | Regression, Logistic | .301 | This p-value is from the same regression as the high risk vs. top 10% risk contrast (analysis 1). | Superiority |
|
| Asian |
|
|
| Black or African American |
|
|
| Native Hawaiian or Other Pacific Islander |
|
|
| White |
|
|
| Unknown |
|
|
|
| Male |
|
|
|
| Not Hispanic or Latino |
|
|
| Unable to obtain |
|
|