Background: While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly... Show moreBackground: While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood. Methods: We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed. Results: Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73-0.78]) and validation (0.64 [interquartile range 0.60-0.67], P<0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59-0.90]) than distantly related cohorts (0.59 [interquartile range 0.43-0.73], P=0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation. Conclusions: There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making. Show less
Schmidt, D.E.; Edslev, P.W.; Heitink-Polle, K.M.J.; Mertens, B.; Bruin, M.C.A.; Kapur, R.; ... ; Haas, M. de 2020
Background Childhood immune thrombocytopenia (ITP) is an autoimmune bleeding disorder. The prognosis (transient, persistent, or chronic ITP) remains difficult to predict. The morbidity is most... Show moreBackground Childhood immune thrombocytopenia (ITP) is an autoimmune bleeding disorder. The prognosis (transient, persistent, or chronic ITP) remains difficult to predict. The morbidity is most pronounced in children with persistent and chronic ITP. Clinical characteristics are associated with ITP outcomes, but there are no validated multivariate prediction models.Objective Development and external validatation of the Childhood ITP Recovery Score to predict transient versus persistent ITP in children with newly diagnosed ITP.Methods Patients with a diagnosis platelet count <= 20 x 10(9)/L and age below 16 years were included from two prospective multicenter studies (NOPHO ITP study, N = 377 [development cohort]; TIKI trial, N = 194 [external validation]). The primary outcome was transient ITP (complete recovery with platelets >= 100 x 10(9)/L 3 months after diagnosis) versus persistent ITP. Age, sex, mucosal bleeding, preceding infection/vaccination, insidious onset, and diagnosis platelet count were used as predictors.Results In external validation, the score predicted transient versus persistent ITP at 3 months follow-up with an area under the receiver operating characteristic curve of 0.71. In patients predicted to have a high chance of recovery, we observed 85%, 90%, and 95% recovered 3, 6, and 12 months after the diagnosis. For patients predicted to have a low chance of recovery, this was 32%, 46%, and 71%. The score also predicted cessation of bleeding symptoms and the response to intravenous immunoglobulins (IVIg).Conclusion The Childhood ITP Recovery Score predicts prognosis and may be useful to individualize clinical management. In future research, the additional predictive value of biomarkers can be compared to this score. A risk calculator is available (). Show less
Klok, F.A.; Dzikowska-Diduch, O.; Kostrubiec, M.; Vliegen, H.W.; Pruszczyk, P.; Hasenfuss, G.; ... ; Lankeit, M. 2016
