Prediction of Multidrug-resistant Bacterial Infection in Patients with Cirrhosis
Prediction of Multidrug-resistant Bacterial Infection in Patients with Cirrhosis
The aim of this study is to identify predictive factors of infections caused by multidrug-resistant organisms in patients with cirrhosis and to develop and validate (internally and externally) a predictive model that might be useful to use in clinical settings to stratify the risk and lead clinical decision-making strategies.
Background Bacterial infections are currently recognized as a surrogate for the final stage of chronic liver disease. The relevance of bacterial infections as a prognostic factor has been clearly stated in a meta-analysis that found that bacterial infections increase mortality four-fold in this population, considering 30% of patients die within one month, and another 30% die one year after these infections are diagnosed.
Patients with cirrhosis are particularly prone to be colonized or infected by multidrug-resistant organisms because they are frequently admitted to hospitals, they require admissions to critical care units and receive organ support, they are exposed to invasive procedures and they frequently receive therapeutic and prophylactic antibiotics.
Prior studies in patients with cirrhosis mainly described risk factors of infections by multidrug-resistant organisms infections, without providing a risk assessment. Therefore, it would be desirable to count on tools to identify which patients are at higher risk than others of presenting a multidrug-resistant organism infection, such as a predictive model that might help practitioners to select the empirical antibiotic treatment.
Objectives To identify easily accessible predictors of multidrug resistance in adult patients with cirrhosis at the time of bacterial infection diagnosis or suspicion.
To develop and validate (goodness of fit, discrimination, calibration, and diagnostic performance) predictive models of multidrug resistance in adult patients with cirrhosis at the time of bacterial infection
Setting and Study design
Cross-sectional study using data from two prospective cohort studies:
Population As mentioned, the objective of this research is to develop and validate a predictive model of multidrug resistance. However, multidrug resistance is observable only in episodes of culture-positive bacterial infections, and it is known that approximately 50% of all bacterial infections taken together are culture-negative. Since the predictive score is expected to be used bedside at the time of the suspicion or confirmation of the bacterial infection, but prior to the results of the cultures, all patients (culture-positive and culture-negative) from the aforementioned studies will be used to develop and internally and externally validate the model.
Outcome variable The outcome will be infection by multidrug-resistant organisms defined in both studies according to the current recommendation: an infection caused by an organism with acquired resistance to at least one antibiotic of three different families. All episodes of infection with a culture yielding a multidrug-resistant organism will be classified as 1. All culture-negative episodes of infection together with all culture-positive infections without multidrug-resistant organisms will be classified as 0 in the principal analysis. In the secondary analysis for validation, all culture-negative episodes of infections will be excluded.
Sampling and sample size calculation In both studies, sampling was consecutive. Regarding sample size calculation for the development of the model, the rule of thumb will be used. Even though there are more modern proposals to approach the sample size to generate predictive models, the lack of prior literature in this field prevents the research group from having the necessary information that this new method requires. Therefore, as the dataset that will be used to develop the predictive model contains a total of 253 events (infection by multidrug-resistant organisms infection) a model starting with 13 to 25 parameters could be explored, considering the rule of thumb proposes to include 10 to 20 events per parameter (EPP).
The prevalence of multidrug-resistant organisms is approximately 20% (253/1302) in the development dataset. A parsimonious predictive model with around 15 potential parameters will be used. The sample size was calculated to keep a MAPE (mean absolute prediction error) of 0.05, a shrinkage factor<10%, and a low optimism of 0.05. The expected Cox-Snell R2 of 0.1 was used. The following are the sample size estimations using Riley's approach in 4 steps to estimate sample size: 1. Total sample size 246 (3.28 EPP) to estimate the independent parameter with adequate precision; 2. Total sample size 860 (14.3 EPP) to keep predictive precision (12 because it is the higher number of parameters allowed, https://mvansmeden.shinyapps.io/BeyondEPV/ ); 3. Total sample size 1274 (16.99 EPP) to minimize overfitting; and 4. Total sample size 443 (5.91 EPP) to minimize optimism. The higher total sample size of 1274 was chosen to fulfill all 4 criteria. The sample size was estimated using Ensor and Riley's pmsampsize for STATA.
Regarding sample size for external validation, the literature is more conflicting on how many events and no events are necessary. Given that it is not easy to count with a database to perform external validation, and that some experts suggest that at least 100 events and no events are necessary to do so, the validation dataset was considered adequate since it contains 103 events and a significantly greater number of no-events.
Descriptive and exploratory statistics In both studies, the prevalence of multidrug-resistant organisms infections will be presented with 95% confidence intervals (CI) and will be estimated over the entire population (culture-positive plus culture-negative infections) and also only over the subgroup of patients with culture-positive infections.
The distribution of potential predictors and the outcome will be presented for both cohorts using mean and standard deviations (DS) or median and 25th-75th percentiles for numerical variables, and absolute numbers and percentages for categorical variables.
The prevalence of infections by multidrug-resistant organisms varies in different countries, regions, and even within the same country and medical centers. The frequency of multidrug-resistant organisms will be presented for each country and descriptive statistics will be provided. This is a proxy of multidrug-resistant organisms' frequency in different settings, and hence potentially may be related to the probability of getting a multidrug-resistant organisms infection. Since the aim was to develop and validate a predictive model that can be used in any setting, countries will not be explored as predictors. This potential limitation in the predictive model enhances the possibility of developing a useful model that is not restricted to the countries included in this study.
Development of the predictive model. All patients with complete information (complete case analysis) regarding the study outcome and potential predictors included in the development dataset will be included in the development and internal validation of the predictive model (Piano et al, DOI: 10.1053/j.gastro.2018.12.00, N= 1,302). The following candidate predictors of MDRO infections will be pre-selected by the research group according to prior publications and knowledge in the field and expert opinion: age, sex, norfloxacin prophylaxis, rifaximin_treatment, beta_blockers, prior invasive_procedures, MELD-Na, albumin, intensive care unit admission, acute-on-chronic liver failure, ACLF, ascites, encephalopathy, systemic inflammatory response syndrome, vasopressors, pneumonia, urinary tract infection, spontaneous bacterial peritonitis.
The association between each potential predictor and the outcome of interest will b evaluated with logistic regression models. Odds ratios (OR) with 95% CI will be presented. Pre-specified relevant interactions between pairs of candidate predictors which are expected to occur according to prior publications or knowledge in the field will be explored and included in the model. In case there is no linear association between continuous predictors and the log odds of multidrug-resistant organisms infection, multivariable fractional polynomials (MFP). For this purpose, the STATA mfp command will be used, which performs a multivariable logistic regression with backward stepwise selection for final predictor selection, applying Akaike's Information Criteria (AIC) for the removal of variables, together with the generation of MFPs. The AIC will be approached in STATA using backward elimination with a p-value of 0.157 as a proxy since STATA would not perform it automatically. The following predictors which are considered to have great clinical relevance will be forced to be in the final model: therapeutic antibiotic use over the last three months, and type of infection. Homoscedasticity will be explored graphically with a Scatter plot of standardized Pearson residuals over the predicted probability of infection by multidrug-resistant organisms. The collinearity of the predictors will be explored with dot plots and correlation coefficients. Extreme values will be explored using Cook's distance.
The full linear predictor of the model will be presented. By full linear predictor, it is meant what is highlighted in the following equation: Logit (pi)= A + B1X1 + B2X2 +......+BnXn.. Presenting the full linear predictor is a key element for external validation of the developed model. All coefficients will be presented with their standard errors, ORs (95 CI%), and p values. The individual linear predictor will be summarized with mean and SD or median and percentiles 25%-75%, and ranges. Uniform shrinkage will be applied to the full linear predictor to correct for optimism, and presented as well, like this: A1 + [ S* (B1X1 + B2X2 +......+BnXn.), where A1 is the intercept re-estimated to ensure calibration-in- the-large, and S is the shrinkage factor. The uniform shrinkage factor is the optimism-adjusted calibration slope calculated during bootstrapping internal validation (explained below).
The goodness of fit of the developed model will be explored with the following estimands:
Brier´s score, which ranges from 0 to 0.25 (the lowest the better the model fits).
Nagelkerke´s R2 (the higher the better the model fits). Akaike's information criteria (the lowest the better)
Apparent validation Apparent validation will be explored in the original dataset, without resampling, and will be evaluated with calibration and discrimination measures. Calibration will be evaluated with observed/expected ratio, calibration plots, calibration in the large (CITL), and calibration slope. Discrimination will be evaluated by estimating the area under the ROC curve (also named C-index or C-statistic when applying logistic regression). Additionally, the diagnostic performance will be evaluated, reporting the sensitivity, specificity, positive predictive, and negative predictive values of different cutoff points or risks of multidrug-resistant bacterial infection.
Internal Validation
For internal validation, bootstrapping will be applied using 300 samples of the same size drawn with replacement from the development dataset. This strategy mimics sampling from the target population by taking the original dataset and randomly selecting units of analysis (i.e patients) with replacement until a dataset of the same size is obtained. With every bootstrap sample, a model is generated using the same methods as described above, and its performance is evaluated, using the original dataset. The following steps are applied for the purpose of internal validation using bootstrap :
Produce a bootstrap sample Develop a model in the bootstrap sample (using the same methods as originally done) Calculate apparent performance (in the bootstrap sample) and test performance (in the original data) Calculate optimism for each estimator (apparent - test) Repeat steps prior steps 300 times to obtain the distribution of optimism estimates Calculate average optimism, and adjust original model estimates: AUC, calibration slope and CITL.
The reliability of predictors selection using the same bootstrapping procedure will be presented. The reliability of prediction selection is the percentage of bootstrapping samples in which a particular predictor remains in the final predictive model according to the same automatic procedure used to construct the model, divided by the total number of bootstrapping samples. Predictors reliability over 50% are considered stable and this is strong evidence that should be included in the final model.
External validation of the predictive model All patients from the external validation dataset (Marciano et al, NCT03919032, N=472) with complete data on the predictors and the outcome will be used for external validation, with the intention to explore the transportability of the predictive model. The full shrunken linear predictor (i.e beta coefficients and intercept) of the developed model will be used. Calibration and discrimination will be estimated in the external validation of the predictive model. Calibration will be evaluated with observed/expected ratio, calibration plots, calibration in the large (CITL), and calibration slope. Discrimination will be evaluated by estimating the area under the ROC curve.
Subgroup analysis Validation evaluation will be repeated as a secondary analysis considering only patients with culture-positive infections.
Management of missing data The proportion of missing data is expected to be low considering the information available from both cohorts' publications or communications in scientific meetings. Since the predominant missing mechanism may be Missing Completely at Random, complete cases as a random sample of all the observations will be considered. Under this assumption, all data analysis will be performed as a complete case analysis. The number of missing values will be reported.
Inclusion Criteria:
Exclusion Criteria:
Buenos Aires, Buenos Aires 1643, Argentina