After mild traumatic brain injury (mTBI), a substantial proportion of individuals do not fully recover on the Glasgow Outcome Scale Extended (GOSE) or experience persistent post-concussion symptoms... Show moreAfter mild traumatic brain injury (mTBI), a substantial proportion of individuals do not fully recover on the Glasgow Outcome Scale Extended (GOSE) or experience persistent post-concussion symptoms (PPCS). We aimed to develop prognostic models for the GOSE and PPCS at 6 months after mTBI and to assess the prognostic value of different categories of predictors (clinical variables; questionnaires; computed tomography [CT]; blood biomarkers). From the Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI) study, we included participants aged 16 or older with Glasgow Coma Score (GCS) 13-15. We used ordinal logistic regression to model the relationship between predictors and the GOSE, and linear regression to model the relationship between predictors and the Rivermead Post-concussion Symptoms Questionnaire (RPQ) total score. First, we studied a pre-specified Core model. Next, we extended the Core model with other clinical and sociodemographic variables available at presentation (Clinical model). The Clinical model was then extended with variables assessed before discharge from hospital: early post-concussion symptoms, CT variables, biomarkers, or all three categories (extended models). In a subset of patients mostly discharged home from the emergency department, the Clinical model was extended with 2-3-week post-concussion and mental health symptoms. Predictors were selected based on Akaike's Information Criterion. Performance of ordinal models was expressed as a concordance index (C) and performance of linear models as proportion of variance explained (R2). Bootstrap validation was used to correct for optimism. We included 2376 mTBI patients with 6-month GOSE and 1605 patients with 6-month RPQ. The Core and Clinical models for GOSE showed moderate discrimination (C = 0.68 95% confidence interval 0.68 to 0.70 and C = 0.70[0.69 to 0.71], respectively) and injury severity was the strongest predictor. The extended models had better discriminative ability (C = 0.71[0.69 to 0.72] with early symptoms; 0.71[0.70 to 0.72] with CT variables or with blood biomarkers; 0.72[0.71 to 0.73] with all three categories). The performance of models for RPQ was modest (R2 = 4% Core; R2 = 9% Clinical), and extensions with early symptoms increased the R2 to 12%. The 2-3-week models had better performance for both outcomes in the subset of participants with these symptoms measured (C = 0.74 [0.71 to 0.78] vs. C = 0.63[0.61 to 0.67] for GOSE; R2 = 37% vs. 6% for RPQ). In conclusion, the models based on variables available before discharge have moderate performance for the prediction of GOSE and poor performance for the prediction of PPCS. Symptoms assessed at 2-3 weeks are required for better predictive ability of both outcomes. The performance of the proposed models should be examined in independent cohorts. Show less
Prognostic prediction of traumatic brain injury (TBI) in patients is crucial in clinical decision and health care policy making. This study aimed to develop and validate prediction models for in... Show morePrognostic prediction of traumatic brain injury (TBI) in patients is crucial in clinical decision and health care policy making. This study aimed to develop and validate prediction models for in-hospital mortality after severe traumatic brain injury (sTBI). We developed and validated logistic regression (LR), LASSO regression, and machine learning (ML) algorithms including support vector machines (SVM) and XGBoost models. Fifty-four candidate predictors were included. Model performance was expressed in terms of discrimination (C-statistic) and calibration (intercept and slope). For model development, 2804 patients with sTBI in the Collaborative European NeuroTrauma Effectiveness Research in TBI (CENTER-TBI) China Registry study were included. External validation was performed in 1113 patients with sTBI in the CENTER-TBI European Registry study. XGBoost achieved high discrimination in mortality prediction, and it outperformed logistic and LASSO regression. The XGBoost model established in this study also outperformed prediction models currently available, including the International Mission for Prognosis and Analysis of Clinical Trials (IMPACT) core and International Mission for Prognosis and Analysis of Clinical Trials (CRASH) basic models. When including 54 variables, XGBoost and SVM reached C-statistics of 0.87 (95% confidence interval [CI]: 0.81-0.92) and 0.85 (95% CI: 0.79-0.90) at internal validation, and 0.88 (95% CI: 0.87-0.88) and 0.86 (95% CI: 0.85-0.87) at external validation, respectively. A simplified version of XGBoost and SVM using 26 variables selected by recursive feature elimination (RFE) reached C-statistics of 0.87 (95% CI: 0.82-0.92) and 0.86 (95% CI: 0.80-0.91) at internal validation, and 0.87 (95% CI: 0.87-0.88) and 0.87 (95% CI: 0.86-0.87) at external validation, respectively. However, when the number of variables included decreased, the difference between ML and LR diminished. All the prediction models can be accessed via a web-based calculator. Glasgow Coma Scale (GCS) score, age, pupillary light reflex, Injury Severity Score (ISS) for brain region, and the presence of acute subdural hematoma were the five strongest predictors for mortality prediction. The study showed that ML techniques such as XGBoost may capture information hidden in demographic and clinical predictors of patients with sTBI and yield more precise predictions compared with LR approaches. Show less
Ramspek, C.L.; Boekee, R.; Evans, M.; Heimburger, O.; Snead, C.M.; Caskey, F.J.; ... ; EQUAL Study Investigators 2022
Introduction: Predicting the timing and occurrence of kidney replacement therapy (KRT), cardiovascular events, and death among patients with advanced chronic kidney disease (CKD) is clinically... Show moreIntroduction: Predicting the timing and occurrence of kidney replacement therapy (KRT), cardiovascular events, and death among patients with advanced chronic kidney disease (CKD) is clinically useful and relevant. We aimed to externally validate a recently developed CKD G4thorn risk calculator for these outcomes and to assess its potential clinical impact in guiding vascular access placement.Methods: We included 1517 patients from the European Quality (EQUAL) study, a European multicentre prospective cohort study of nephrology-referred advanced CKD patients aged $65 years. Model performance was assessed based on discrimination and calibration. Potential clinical utility for timing of referral for vascular access placement was studied with diagnostic measures and decision curve analysis (DCA).Results: The model showed a good discrimination for KRT and "death after KRT," with 2-year concordance (C) statistics of 0.74 and 0.76, respectively. Discrimination for cardiovascular events (2-year C-statistic: 0.70) and overall death (2-year C-statistic: 0.61) was poorer. Calibration was fairly accurate. Decision curves illustrated that using the model to guide vascular access referral would generally lead to less unused arteriovenous fistulas (AVFs) than following estimated glomerular filtration rate (eGFR) thresholds.Conclusion: This study shows moderate to good predictive performance of the model in an older cohort of nephrology-referred patients with advanced CKD. Using the model to guide referral for vascular access placement has potential in combating unnecessary vascular surgeries. Show less
Ramspek, C.L.; Teece, L.; Snell, K.I.E.; Evans, M.; Riley, R.D.; Smeden, M. van; ... ; Diepen, M. van 2021
Background: External validation of prognostic models is necessary to assess the accuracy and generalizability of the model to new patients. If models are validated in a setting in which competing... Show moreBackground: External validation of prognostic models is necessary to assess the accuracy and generalizability of the model to new patients. If models are validated in a setting in which competing events occur, these competing risks should be accounted for when comparing predicted risks to observed outcomes. Methods: We discuss existing measures of calibration and discrimination that incorporate competing events for time-to-event models. These methods are illustrated using a clinical-data example concerning the prediction of kidney failure in a population with advanced chronic kidney disease (CKD), using the guideline-recommended Kidney Failure Risk Equation (KFRE). The KFRE was developed using Cox regression in a diverse population of CKD patients and has been proposed for use in patients with advanced CKD in whom death is a frequent competing event. Results: When validating the 5-year KFRE with methods that account for competing events, it becomes apparent that the 5-year KFRE considerably overestimates the real-world risk of kidney failure. The absolute overestimation was 10%age points on average and 29%age points in older high-risk patients. Conclusions: It is crucial that competing events are accounted for during external validation to provide a more reliable assessment the performance of a model in clinical settings in which competing risks occur. Show less
Chalkou, K.; Steyerberg, E.; Egger, M.; Manca, A.; Pellegrini, F.; Salanti, G. 2021
Treatment effects vary across different patients, and estimation of this variability is essential for clinical decision-making. We aimed to develop a model estimating the benefit of alternative... Show moreTreatment effects vary across different patients, and estimation of this variability is essential for clinical decision-making. We aimed to develop a model estimating the benefit of alternative treatment options for individual patients, extending a risk modeling approach in a network meta-analysis framework. We propose a two-stage prediction model for heterogeneous treatment effects by combining prognosis research and network meta-analysis methods where individual patient data are available. In the first stage, a prognostic model to predict the baseline risk of the outcome. In the second stage, we use the baseline risk score from the first stage as a single prognostic factor and effect modifier in a network meta-regression model. We apply the approach to a network meta-analysis of three randomized clinical trials comparing the relapses in Natalizumab, Glatiramer Acetate, and Dimethyl Fumarate, including 3590 patients diagnosed with relapsing-remitting multiple sclerosis. We find that the baseline risk score modifies the relative and absolute treatment effects. Several patient characteristics, such as age and disability status, impact the baseline risk of relapse, which in turn moderates the benefit expected for each of the treatments. For high-risk patients, the treatment that minimizes the risk of relapse in 2 years is Natalizumab, whereas Dimethyl Fumarate might be a better option for low-risk patients. Our approach can be easily extended to all outcomes of interest and has the potential to inform a personalized treatment approach. Show less
Background: SARS-CoV-2 is straining health care systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate patients who... Show moreBackground: SARS-CoV-2 is straining health care systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate patients who require hospitalization from those who do not. The COVID-19 vulnerability (C-19) index, a model that predicts which patients will be admitted to hospital for treatment of pneumonia or pneumonia proxies, has been developed and proposed as a valuable tool for decision-making during the pandemic. However, the model is at high risk of bias according to the "prediction model risk of bias assessment" criteria, and it has not been externally validated.Objective: The aim of this study was to externally validate the C-19 index across a range of health care settings to determine how well it broadly predicts hospitalization due to pneumonia in COVID-19 cases.Methods: We followed the Observational Health Data Sciences and Informatics (OHDSI) framework for external validation to assess the reliability of the C-19 index. We evaluated the model on two different target populations, 41,381 patients who presented with SARS-CoV-2 at an outpatient or emergency department visit and 9,429,285 patients who presented with influenza or related symptoms during an outpatient or emergency department visit, to predict their risk of hospitalization with pneumonia during the following 0-30 days. In total, we validated the model across a network of 14 databases spanning the United States, Europe, Australia, and Asia.Results: The internal validation performance of the C-19 index had a C statistic of 0.73, and the calibration was not reported by the authors. When we externally validated it by transporting it to SARS-CoV-2 data, the model obtained C statistics of 0.36, 0.53 (0.473-0.584) and 0.56 (0.488-0.636) on Spanish, US, and South Korean data sets, respectively. The calibration was poor, with the model underestimating risk. When validated on 12 data sets containing influenza patients across the OHDSI network, the C statistics ranged between 0.40 and 0.68.Conclusions: Our results show that the discriminative performance of the C-19 index model is low for influenza cohorts and even worse among patients with COVID-19 in the United States, Spain, and South Korea. These results suggest that C-19 should not be used to aid decision-making during the COVID-19 pandemic. Our findings highlight the importance of performing external validation across a range of settings, especially when a prediction model is being extrapolated to a different population. In the field of prediction, extensive validation is required to create appropriate trust in a model. Show less
Mikolic, A.; Polinder, S.; Steyerberg, E.W.; Helmrich, I.R.A.R.; Giacino, J.T.; Maas, A.I.R.; ... ; CENTER TBI Study Pa 2020
The majority of traumatic brain injuries (TBIs) are categorized as mild, according to a baseline Glasgow Coma Scale (GCS) score of 13-15. Prognostic models that were developed to predict functional... Show moreThe majority of traumatic brain injuries (TBIs) are categorized as mild, according to a baseline Glasgow Coma Scale (GCS) score of 13-15. Prognostic models that were developed to predict functional outcome and persistent post-concussive symptoms (PPCS) after mild TBI have rarely been externally validated. We aimed to externally validate models predicting 3-12-month Glasgow Outcome Scale Extended (GOSE) or PPCS in adults with mild TBI. We analyzed data from the Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI) project, which included 2862 adults with mild TBI, with 6-month GOSE available for 2374 and Rivermead Post-Concussion Symptoms Questionnaire (RPQ) results available for 1605 participants. Model performance was evaluated based on calibration (graphically and characterized by slope and intercept) and discrimination (C-index). We validated five published models for 6-month GOSE and three for 6-month PPCS scores. The models used different cutoffs for outcome and some included symptoms measured 2 weeks post-injury. Discriminative ability varied substantially (C-index between 0.58 and 0.79). The models developed in the Corticosteroid Randomisation After Significant Head Injury (CRASH) trial for prediction of GOSE <5 discriminated best (C-index 0.78 and 0.79), but were poorly calibrated. The best performing models for PPCS included 2-week symptoms (C-index 0.75 and 0.76). In conclusion, none of the prognostic models for early prediction of GOSE and PPCS has both good calibration and discrimination in persons with mild TBI. In future studies, prognostic models should be tailored to the population with mild TBI, predicting relevant end-points based on readily available predictors. Show less
Uveal melanoma (UM) is fatal in -50% of patients as a result of disseminated disease. This study aims to externally validate the Liverpool Uveal Melanoma Prognosticator Online V3 (LUMPO3) to... Show moreUveal melanoma (UM) is fatal in -50% of patients as a result of disseminated disease. This study aims to externally validate the Liverpool Uveal Melanoma Prognosticator Online V3 (LUMPO3) to determine its reliability in predicting survival after treatment for choroidal melanoma when utilizing external data from other ocular oncology centers. Anonymized data of 1836 UM patients from seven international ocular oncology centers were analyzed with LUMPO3 to predict the 10-year survival for each patient in each external dataset. The analysts were masked to the patient outcomes. Model predictions were sent to an independent statistician to evaluate LUMPO3's performance using discrimination and calibration methods. LUMPO3's ability to discriminate between UM patients who died of metastatic UM and those who were still alive was fair-to-good, with C-statistics ranging from 0.64 to 0.85 at year 1. The pooled estimate for all external centers was 0.72 (95% confidence interval: 0.68 to 0.75). Agreement between observed and predicted survival probabilities was generally good given differences in case mix and survival rates between different centers. Despite the differences between the international cohorts of patients with primary UM, LUMPO3 is a valuable tool for predicting all-cause mortality in this disease when using data from external centers. Show less
Houweling, T.A.J.; Klaveren, D. van; S. das; Azad, K.; Tripathy, P.; Manandhar, D.; ... ; Costello, A. 2019
