Background: Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models... Show moreBackground: Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models. Methods: We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers. Results: Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models. Conclusion: Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating. Show less
Aarts, G.; Benda-Beckmann, A.M. von der; Lucke, K.; Sertlek, H.O.; Bemmelen, R. van; Geelhoed, S.C.V.; ... ; Kirkwood, R. 2016
The European Commission introduced impact assessment (IA) in 2002 following recommendations from the Mandelkern group on Better Regulation. The basic rationale of IA is that proposals must be... Show moreThe European Commission introduced impact assessment (IA) in 2002 following recommendations from the Mandelkern group on Better Regulation. The basic rationale of IA is that proposals must be prepared on the basis of an analysis of whether regulatory intervention is needed and whether it is appropriate to intervene at the EU level. An IA report, published together with the proposal, summarises the results and highlights the trade-offs between the potential economic, social and environmental impacts associated with various policy options. Nowadays, IA is required for all major proposals. It is also increasingly being used throughout the whole legislative process as the European Parliament and the Council of Ministers are implementing their respective commitments to IA. The thesis starts out by developing a typology of possible uses of impact assessment in EU lawmaking, which serves as a framework of reference for the subsequent analysis of the development of EU IA. The idea is explored that the three âco-legislatorsâ are not the only relevant actors in IA processes; national institutions, advisory bodies, review institutions, regulatory bodies, private co-actors and third-country actors all play their part. Case-studies on the new chemicals regulation (REACH), the thematic strategy on air quality (CAFE), the pre-packaging directive and the data retention directive lend empirical backing to the theoretical claims. The thesis arrives at the conclusion that IA is becoming a self-standing normative force in EU lawmaking by contrasting EU IA as 'soft constitutional law' and EU IA as 'meta-regulation'. Show less
