Introduction: In patients with acute respiratory distress syndrome (ARDS), the PaO2/FiO(2) ratio at the time of ARDS diagnosis is weakly associated with mortality. We hypothesized that setting a... Show moreIntroduction: In patients with acute respiratory distress syndrome (ARDS), the PaO2/FiO(2) ratio at the time of ARDS diagnosis is weakly associated with mortality. We hypothesized that setting a PaO2/FiO(2) threshold in 150 mm Hg at 24 h from moderate/severe ARDS diagnosis would improve predictions of death in the intensive care unit (ICU). Methods: We conducted an ancillary study in 1303 patients with moderate to severe ARDS managed with lung-protective ventilation enrolled consecutively in four prospective multicenter cohorts in a network of ICUs. The first three cohorts were pooled (n = 1000) as a testing cohort; the fourth cohort (n = 303) served as a confirmatory cohort. Based on the thresholds for PaO2/FiO(2) (150 mm Hg) and positive end-expiratory pressure (PEEP) (10 cm H2O), the patients were classified into four possible subsets at baseline and at 24 h using a standardized PEEP-FiO(2) approach: (I) PaO2/FiO(2) >= 150 at PEEP < 10, (II) PaO2/FiO(2) > 150 at PEEP >= 10, (III) PaO2/FiO(2) < 150 at PEEP < 10, and (IV) PaO2/FiO(2) < 150 at PEEP >= 10. Primary outcome was death in the ICU. Results: ICU mortalities were similar in the testing and confirmatory cohorts (375/1000, 37.5% vs. 112/303, 37.0%, respectively). At baseline, most patients from the testing cohort (n = 792/1000, 79.2%) had a PaO2/FiO(2) < 150, with similar mortality among the four subsets (p = 0.23). When assessed at 24 h, ICU mortality increased with an advance in the subset: 17.9%, 22.8%, 40.0%, and 49.3% (p < 0.0001). The findings were replicated in the confirmatory cohort (p < 0.0001). However, independent of the PEEP levels, patients with PaO2/FiO(2) < 150 at 24 h followed a distinct 30-day ICU survival compared with patients with PaO2/FiO(2) >= 150 (hazard ratio 2.8, 95% CI 2.2-3.5, p < 0.0001). Conclusions: Subsets based on PaO2/FiO(2) thresholds of 150 mm Hg assessed after 24 h of moderate/severe ARDS diagnosis are clinically relevant for establishing prognosis, and are helpful for selecting adjunctive therapies for hypoxemia and for enrolling patients into therapeutic trials. Show less
Huinink, S.T.; Jong, D.C. de; Nieboer, D.; Thomassen, D.; Steyerberg, E.W.; Dijkgraaf, M.G.W.; ... ; Vries, A.C. de 2022
Background Anti-tumor necrosis factor (TNF) therapy is effective for the treatment of Crohn's disease. Cessation may be considered in patients with a low risk of relapse. We aimed to externally... Show moreBackground Anti-tumor necrosis factor (TNF) therapy is effective for the treatment of Crohn's disease. Cessation may be considered in patients with a low risk of relapse. We aimed to externally validate and update our previously developed prediction model to estimate the risk of relapse after cessation of anti-TNF therapy. Methods We performed a retrospective cohort study in 17 Dutch hospitals. Crohn's disease patients in clinical, biochemical or endoscopic remission were included after anti-TNF cessation. Primary outcome was a relapse necessitating treatment. Discrimination and calibration of the previously developed model were assessed. After external validation, the model was updated. The performance of the updated prediction model was assessed in internal-external validation and by using decision curve analysis. Results 486 patients were included with a median follow-up of 1.7 years. Relapse rates were 35 and 54% after 1 and 2 years. At external validation, the discriminative ability of the prediction model was equal to that found at the development of the model [c-statistic 0.58 (95% confidence interval (CI) 0.54-0.62)], though the model was not well-calibrated on our cohort [calibration slope: 0.52 (0.28-0.76)]. After an update, a c-statistic of 0.60 (0.58-0.63) and calibration slope of 0.89 (0.69-1.09) were reported in internal-external validation. Conclusion Our previously developed and updated prediction model for the risk of relapse after cessation of anti-TNF in Crohn's disease shows reasonable performance. The use of the model may support clinical decision-making to optimize patient selection in whom anti-TNF can be withdrawn. Clinical validation is ongoing in a prospective randomized trial. Show less
Simple Summary The objective was to develop and internally validate a predictive model based on preoperative predictors, including geriatric characteristics, for severe postoperative complications... Show moreSimple Summary The objective was to develop and internally validate a predictive model based on preoperative predictors, including geriatric characteristics, for severe postoperative complications after elective surgery for stage I-III CRC in patients >= 70 years. Potential predictors included demographics, comorbidity, tumour location, activities of daily living (ADL), history of falls, malnutrition, risk factors for delirium, use of mobility aid and polypharmacy. The least absolute shrinkage and selection operator (LASSO) method was used for predictor selection and prediction model building. A geriatric model that included gender, previous DVT or pulmonary embolism, COPD/asthma/emphysema, rectal cancer, the use of a mobility aid, ADL assistance, previous delirium and polypharmacy showed satisfactory discrimination with an AUC of 0.69 (95% CI 0.73-0.64); the AUC for the optimism corrected model was 0.65. An eight-item colorectal geriatric model (GerCRC) was developed. After external validation, this risk model has the potential to be used for preoperative (shared) decision-making. Introduction Older patients have an increased risk of morbidity and mortality after colorectal cancer (CRC) surgery. Existing CRC surgical prediction models have not incorporated geriatric predictors, limiting applicability for preoperative decision-making. The objective was to develop and internally validate a predictive model based on preoperative predictors, including geriatric characteristics, for severe postoperative complications after elective surgery for stage I-III CRC in patients >= 70 years. Patients and Methods: A prospectively collected database contained 1088 consecutive patients from five Dutch hospitals (2014-2017) with 171 severe complications (16%). The least absolute shrinkage and selection operator (LASSO) method was used for predictor selection and prediction model building. Internal validation was done using bootstrapping. Results: A geriatric model that included gender, previous DVT or pulmonary embolism, COPD/asthma/emphysema, rectal cancer, the use of a mobility aid, ADL assistance, previous delirium and polypharmacy showed satisfactory discrimination with an AUC of 0.69 (95% CI 0.73-0.64); the AUC for the optimism corrected model was 0.65. Based on these predictors, the eight-item colorectal geriatric model (GerCRC) was developed. Conclusion: The GerCRC is the first prediction model specifically developed for older patients expected to undergo CRC surgery. Combining tumour- and patient-specific predictors, including geriatric predictors, improves outcome prediction in the heterogeneous older population. Show less
Background: SARS-CoV-2 is straining health care systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate patients who... Show moreBackground: SARS-CoV-2 is straining health care systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate patients who require hospitalization from those who do not. The COVID-19 vulnerability (C-19) index, a model that predicts which patients will be admitted to hospital for treatment of pneumonia or pneumonia proxies, has been developed and proposed as a valuable tool for decision-making during the pandemic. However, the model is at high risk of bias according to the "prediction model risk of bias assessment" criteria, and it has not been externally validated.Objective: The aim of this study was to externally validate the C-19 index across a range of health care settings to determine how well it broadly predicts hospitalization due to pneumonia in COVID-19 cases.Methods: We followed the Observational Health Data Sciences and Informatics (OHDSI) framework for external validation to assess the reliability of the C-19 index. We evaluated the model on two different target populations, 41,381 patients who presented with SARS-CoV-2 at an outpatient or emergency department visit and 9,429,285 patients who presented with influenza or related symptoms during an outpatient or emergency department visit, to predict their risk of hospitalization with pneumonia during the following 0-30 days. In total, we validated the model across a network of 14 databases spanning the United States, Europe, Australia, and Asia.Results: The internal validation performance of the C-19 index had a C statistic of 0.73, and the calibration was not reported by the authors. When we externally validated it by transporting it to SARS-CoV-2 data, the model obtained C statistics of 0.36, 0.53 (0.473-0.584) and 0.56 (0.488-0.636) on Spanish, US, and South Korean data sets, respectively. The calibration was poor, with the model underestimating risk. When validated on 12 data sets containing influenza patients across the OHDSI network, the C statistics ranged between 0.40 and 0.68.Conclusions: Our results show that the discriminative performance of the C-19 index model is low for influenza cohorts and even worse among patients with COVID-19 in the United States, Spain, and South Korea. These results suggest that C-19 should not be used to aid decision-making during the COVID-19 pandemic. Our findings highlight the importance of performing external validation across a range of settings, especially when a prediction model is being extrapolated to a different population. In the field of prediction, extensive validation is required to create appropriate trust in a model. Show less
Background: The impact of precision psychiatry for clinical practice has not been systematically appraised. This study aims to provide a comprehensive review of validated prediction models to... Show moreBackground: The impact of precision psychiatry for clinical practice has not been systematically appraised. This study aims to provide a comprehensive review of validated prediction models to estimate the individual risk of being affected with a condition (diagnostic), developing outcomes (prognostic), or responding to treatments (predictive) in mental disorders. Methods: PRISMA/RIGHT/ CHARMS-compliant systematic review of the Web of Science, Cochrane Central Register of Reviews, and Ovid/PsycINFO databases from inception until July 21, 2019 (PROSPERO CRD42019155713) to identify diagnostic/prognostic/predictive prediction studies that reported individualized estimates in psychiatry and that were internally or externally validated or implemented. Random effect meta-regression analyses addressed the impact of several factors on the accuracy of prediction models. Findings: Literature search identified 584 prediction modeling studies, of which 89 were included. 10.4% of the total studies included prediction models internally validated (n = 61), 4.6% models externally validated (n = 27), and 0.2% (n = 1) models considered for implementation. Across validated prediction modeling studies (n = 88), 18.2% were diagnostic, 68.2% prognostic, and 13.6% predictive. The most frequently investigated condition was psychosis (36.4%), and the most frequently employed predictors clinical (69.5%). Unimodal compared to multimodal models (beta = .29, P = .03) and diagnostic compared to prognostic (beta = .84, p < .0001) and predictive (beta = .87, P = .002) models were associated with increased accuracy. Interpretation: To date, several validated prediction models arc available to support the diagnosis and prognosis of psychiatric conditions, in particular, psychosis, or to predict treatment response. Advancements of knowledge are limited by the lack of implementation research in real-world clinical practice. A new generation of implementation research is required to address this translational gap. Show less
In medical research, missing data is common. In acute diseases, such as traumatic brain injury (TBI), even well-conducted prospective studies may suffer from missing data in baseline... Show moreIn medical research, missing data is common. In acute diseases, such as traumatic brain injury (TBI), even well-conducted prospective studies may suffer from missing data in baseline characteristics and outcomes. Statistical models may simply drop patients with any missing values, potentially leaving a selected subset of the original cohort. Imputation is widely accepted by methodologists as an appropriate way to deal with missing data. We aim to provide practical guidance on handling missing data for prediction modeling. We hereto propose a five-step approach, centered around single and multiple imputation: 1) explore the missing data patterns; 2) choose a method of imputation; 3) perform imputation; 4) assess diagnostics of the imputation; and 5) analyze the imputed data sets. We illustrate these five steps with the estimation and validation of the IMPACT (International Mission on Prognosis and Analysis of Clinical Trials in Traumatic Brain Injury) prognostic model in 1375 patients from the CENTER-TBI database, included in 53 centers across 17 countries, with moderate or severe TBI in the prospective European CENTER-TBI study. Future prediction modeling studies in acute diseases may benefit from following the suggested five steps for optimal statistical analysis and interpretation, after maximal effort has been made to minimize missing data. Show less
The role of P-values for null hypothesis testing is under debate. We aim to explore the impact of the significance threshold on estimates for the strengths of associations ("effects") and the... Show moreThe role of P-values for null hypothesis testing is under debate. We aim to explore the impact of the significance threshold on estimates for the strengths of associations ("effects") and the implications for different types of epidemiological research. We consider situations with normal distribution of a true effect, while varying the effect size. We confirm the occurrence of "testimation bias": estimating effect size only if the test was statistically significant leads to exaggerated results. The absolute bias is largest for true effects around 0.7 times the size of the standard error: +220% bias if effects are selected after testing with P < .05, and +335% if tested with P P < .20 (+130%) and larger true effect sizes. We conclude that a lower P-value threshold for declaring statistical significance implies more exaggeration in an estimated effect. This implies that if a low threshold is used, effect size estimation should not be attempted, for example in the context of selecting promising discoveries that need further validation. Confirmatory studies, such as randomized controlled trials, might stick to the 0.05 threshold if adequately powered, while prediction modelling studies should use an even higher threshold, such as 0.2, to avoid strongly biased effect estimates. Show less
Steyerberg, E.W.; Nieboer, D.; Debray, T.P.A.; Houwelingen, H.C. van 2019
Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta... Show moreClinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta-analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. These methods are illustrated in a case study with patients suffering from traumatic brain injury, where we aim to predict 6-month mortality based on individual patient data using meta-analytic techniques (15 studies, n = 11 022 patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions. Show less
