Background: Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models... Show moreBackground: Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models. Methods: We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers. Results: Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models. Conclusion: Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating. Show less
Objective: To assess whether the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and a shorter version of this tool can identify clinical prediction models (CPMs) that perform poorly at... Show moreObjective: To assess whether the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and a shorter version of this tool can identify clinical prediction models (CPMs) that perform poorly at external validation. Study Design and Setting: We evaluated risk of bias (ROB) on 102 CPMs from the Tufts CPM Registry, comparing PROBAST to a short form consisting of six PROBAST items anticipated to best identify high ROB. We then applied the short form to all CPMs in the Registry with at least 1 validation (n = 556) and assessed the change in discrimination (dAUC) in external validation cohorts (n = 1,147). Results: PROBAST classified 98/102 CPMS as high ROB. The short form identified 96 of these 98 as high ROB (98% sensitivity), with perfect specificity. In the full CPM registry, 527 of 556 CPMs (95%) were classified as high ROB, 20 (3.6%) low ROB, and 9 (1.6%) unclear ROB. Only one model with unclear ROB was reclassified to high ROB after full PROBAST assessment of all low and unclear ROB models. Median change in discrimination was significantly smaller in low ROB models (dAUC -0.9%, IQR -6.2-4.2%) compared to high ROB models (dAUC -11.7%, IQR -33.3-2.6%; P < 0.001). Conclusion: High ROB is pervasive among published CPMs. It is associated with poor discriminative performance at validation, supporting the application of PROBAST or a shorter version in CPM reviews. (c) 2021 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license ( http:// creativecommons.org/ licenses/ by- nc- nd/ 4.0/ ) Show less
Roessel, S. van; Strijker, M.; Steyerberg, E.W.; Groen, J.V.; Mieog, J.S.; Groot, V.P.; ... ; Besselink, M.G. 2020
Background: The objective of this study was to validate and update the Amsterdam prediction model including tumor grade, lymph node ratio, margin status and adjuvant therapy, for prediction of... Show moreBackground: The objective of this study was to validate and update the Amsterdam prediction model including tumor grade, lymph node ratio, margin status and adjuvant therapy, for prediction of overall survival (OS) after pancreatoduodenectomy for pancreatic cancer.Methods: We included consecutive patients who underwent pancreatoduodenectomy for pancreatic cancer between 2000 and 2017 at 11 tertiary centers in 8 countries (USA, UK, Germany, Italy, Sweden, the Netherlands, Korea, Australia). Model performance for prediction of OS was evaluated by calibration statistics and Uno's C-statistic for discrimination. Validation followed the TRIPOD statement.Results: Overall, 3081 patients (53% male, median age 66 years) were included with a median OS of 24 months, of whom 38% had N2 disease and 77% received adjuvant chemotherapy. Predictions of 3-year OS were fairly similar to observed OS with a calibration slope of 0.72. Statistical updating of the model resulted in an increase of the C-statistic from 0.63 to 0.65 (95% CI 0.64-0.65), ranging from 0.62 to 0.67 across different countries. The area under the curve for the prediction of 3 -year OS was 0.71 after updating. Median OS was 36, 25 and 15 months for the low, intermediate and high risk group, respectively (P < 0.001).Conclusions: This large international study validated and updated the Amsterdam model for survival prediction after pancreatoduodenectomy for pancreatic cancer. The model incorporates readily available variables with a fairly accurate model performance and robustness across different countries, while novel markers may be added in the future. The risk groups and web-based calculator www pancreascalculaior. corn may facilitate use in daily practice and future trials. (C) 2019 Elsevier Ltd, BASO The Association for Cancer Surgery, and the European Society of Surgical Oncology. All rights reserved. Show less
Background Three tools are currently available to predict the risk of contralateral breast cancer (CBC). We aimed to compare the performance of the Manchester formula, CBCrisk, and PredictCBC in... Show moreBackground Three tools are currently available to predict the risk of contralateral breast cancer (CBC). We aimed to compare the performance of the Manchester formula, CBCrisk, and PredictCBC in patients with invasive breast cancer (BC). Methods We analyzed data of 132,756 patients (4682 CBC) from 20 international studies with a median follow-up of 8.8 years. Prediction performance included discrimination, quantified as a time-dependent Area-Under-the-Curve (AUC) at 5 and 10 years after diagnosis of primary BC, and calibration, quantified as the expected-observed (E/O) ratio at 5 and 10 years and the calibration slope. Results The AUC at 10 years was: 0.58 (95% confidence intervals [CI] 0.57-0.59) for CBCrisk; 0.60 (95% CI 0.59-0.61) for the Manchester formula; 0.63 (95% CI 0.59-0.66) and 0.59 (95% CI 0.56-0.62) for PredictCBC-1A (for settings where BRCA1/2 mutation status is available) and PredictCBC-1B (for the general population), respectively. The E/O at 10 years: 0.82 (95% CI 0.51-1.32) for CBCrisk; 1.53 (95% CI 0.63-3.73) for the Manchester formula; 1.28 (95% CI 0.63-2.58) for PredictCBC-1A and 1.35 (95% CI 0.65-2.77) for PredictCBC-1B. The calibration slope was 1.26 (95% CI 1.01-1.50) for CBCrisk; 0.90 (95% CI 0.79-1.02) for PredictCBC-1A; 0.81 (95% CI 0.63-0.99) for PredictCBC-1B, and 0.39 (95% CI 0.34-0.43) for the Manchester formula. Conclusions Current CBC risk prediction tools provide only moderate discrimination and the Manchester formula was poorly calibrated. Better predictors and re-calibration are needed to improve CBC prediction and to identify low- and high-CBC risk patients for clinical decision-making. Show less
