Real-world evidence (RWE) is increasingly considered in regulatory decision making. When, and to which extent, RWE is considered relevant by regulators likely depends on many factors. This review... Show moreReal-world evidence (RWE) is increasingly considered in regulatory decision making. When, and to which extent, RWE is considered relevant by regulators likely depends on many factors. This review aimed to identify factors that make RWE necessary or desirable to inform regulatory decision making. A scoping review was conducted using literature databases (PubMed, Embase, Emcare, Web of Science, and Cochrane Library) and websites of regulatory agencies, health technology assessment agencies, research institutes, and professional organizations involved with RWE. Articles were included if: (1) they discussed factors or contexts that impact whether RWE could be necessary or desirable in regulatory decision making; (2) focused on pharmacological or biological interventions in humans; and (3) considered decision making in Europe or North America, or without a focus on a specific region. We included 118 articles in the scoping review. Two major themes and six subthemes were identified. The first theme concerns questions addressable with RWE, with subthemes epidemiology and benefit-risk assessment. The second theme concerns contextual factors, with subthemes feasibility, ethical considerations, limitations of available evidence, and disease and treatment-specific aspects. Collectively, these themes encompassed 43 factors influencing the need for RWE in regulatory decisions. Although single factors may not make RWE fully necessary, their cumulative influence could make RWE essential and pivotal in regulatory decision making. This overview contributes to ongoing discussions emphasizing the nuanced interplay of factors influencing the necessity or desirability of RWE to inform regulatory decision making. Show less
Luijken, K.; Wall, B.J.M. van de; Hooft, L.; Leenen, L.P.H.; Houwert, R.M.; Groenwold, R.H.H.; NEXT Study Grp 2022
Purpose: It is challenging to generate and subsequently implement high-quality evidence in surgical practice. A first step would be to grade the strengths and weaknesses of surgical evidence and... Show morePurpose: It is challenging to generate and subsequently implement high-quality evidence in surgical practice. A first step would be to grade the strengths and weaknesses of surgical evidence and appraise risk of bias and applicability. Here, we described items that are common to different risk-of-bias tools. We explained how these could be used to assess comparative operative intervention studies in orthopedic trauma surgery, and how these relate to applicability of results. Methods: We extracted information from the Cochrane risk-of-bias-2 (RoB-2) tool, Risk Of Bias In Non-randomised Studies-of Interventions tool (ROBINS-I), and Methodological Index for Non-Randomized Studies (MINORS) criteria and derived a concisely formulated set of items with signaling questions tailored to operative interventions in orthopedic trauma surgery. Results: The established set contained nine items: population, intervention, comparator, outcome, confounding, missing data and selection bias, intervention status, outcome assessment, and pre-specification of analysis. Each item can be assessed using signaling questions and was explained using good practice examples of operative intervention studies in orthopedic trauma surgery. Conclusion: The set of items will be useful to form a first judgment on studies, for example when including them in a systematic review. Existing risk of bias tools can be used for further evaluation of methodological quality. Additionally, the proposed set of items and signaling questions might be a helpful starting point for peer reviewers and clinical readers. Show less
Pameijer, E.M.; Heus, P.; Damen, J.A.A.; Spijker, R.; Hooft, L.; Ringens, P.J.; ... ; Leeuwen, R. van 2022
The aim of this paper is to summarize all available evidence from systematic reviews, randomized controlled trials (RCTs) and comparative nonrandomized studies (NRS) on the association between... Show moreThe aim of this paper is to summarize all available evidence from systematic reviews, randomized controlled trials (RCTs) and comparative nonrandomized studies (NRS) on the association between nutrition and antioxidant, vitamin, and mineral supplements and the development or progression of age-related macular degeneration (AMD). The Cochrane Database of Systematic Reviews, Cochrane register CENTRAL, MEDLINE and Embase were searched and studies published between January 2015 and May 2021 were included. The certainty of evidence was assessed according to the GRADE methodology. The main outcome measures were development of AMD, progression of AMD, and side effects. We included 7 systematic reviews, 7 RCTs, and 13 NRS. A high consumption of specific nutrients, i.e. beta-carotene, lutein and zeaxanthin, copper, folate, magnesium, vitamin A, niacin, vitamin B6, vitamin C, docosahexaenoic acid, and eicosapentaenoic acid, was associated with a lower risk of progression of early to late AMD (high certainty of evidence). Use of antioxidant supplements and adherence to a Mediterranean diet, characterized by a high consumption of vegetables, whole grains, and nuts and a low consumption of red meat, were associated with a decreased risk of progression of early to late AMD (moderate certainty of evidence). A high consumption of alcohol was associated with a higher risk of developing AMD (moderate certainty of evidence). Supplementary vitamin C, vitamin E, or beta-carotene were not associated with the development of AMD, and supplementary omega-3 fatty acids were not associated with progression to late AMD (high certainty of evidence). Research in the last 35 years included in our overview supports that a high intake of specific nutrients, the use of antioxidant supplements and adherence to a Mediterranean diet decrease the risk of progression of early to late AMD. Show less
While the opportunities of ML and AI in healthcare are promising, the growth of complex data-driven prediction models requires careful quality and applicability assessment before they are applied... Show moreWhile the opportunities of ML and AI in healthcare are promising, the growth of complex data-driven prediction models requires careful quality and applicability assessment before they are applied and disseminated in daily practice. This scoping review aimed to identify actionable guidance for those closely involved in AI-based prediction model (AIPM) development, evaluation and implementation including software engineers, data scientists, and healthcare professionals and to identify potential gaps in this guidance. We performed a scoping review of the relevant literature providing guidance or quality criteria regarding the development, evaluation, and implementation of AIPMs using a comprehensive multi-stage screening strategy. PubMed, Web of Science, and the ACM Digital Library were searched, and AI experts were consulted. Topics were extracted from the identified literature and summarized across the six phases at the core of this review: (1) data preparation, (2) AIPM development, (3) AIPM validation, (4) software development, (5) AIPM impact assessment, and (6) AIPM implementation into daily healthcare practice. From 2683 unique hits, 72 relevant guidance documents were identified. Substantial guidance was found for data preparation, AIPM development and AIPM validation (phases 1-3), while later phases clearly have received less attention (software development, impact assessment and implementation) in the scientific literature. The six phases of the AIPM development, evaluation and implementation cycle provide a framework for responsible introduction of AI-based prediction models in healthcare. Additional domain and technology specific research may be necessary and more practical experience with implementing AIPMs is needed to support further guidance. Show less
Najafabadi, A.H.Z.; Ramspek, C.L.; Dekker, F.W.; Heus, P.; Hooft, L.; Moons, K.G.M.; ... ; Diepen, M. van 2020
Objectives To assess the difference in completeness of reporting and methodological conduct of published prediction models before and after publication of the Transparent Reporting of a... Show moreObjectives To assess the difference in completeness of reporting and methodological conduct of published prediction models before and after publication of the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. Methods In the seven general medicine journals with the highest impact factor, we compared the completeness of the reporting and the quality of the methodology of prediction model studies published between 2012 and 2014 (pre-TRIPOD) with studies published between 2016 and 2017 (post-TRIPOD). For articles published in the post-TRIPOD period, we examined whether there was improved reporting for articles (1) citing the TRIPOD statement, and (2) published in journals that published the TRIPOD statement. Results A total of 70 articles was included (pre-TRIPOD: 32, post-TRIPOD: 38). No improvement was seen for the overall percentage of reported items after the publication of the TRIPOD statement (pre-TRIPOD 74%, post-TRIPOD 76%, 95% CI of absolute difference: -4% to 7%). For the individual TRIPOD items, an improvement was seen for 16 (44%) items, while 3 (8%) items showed no improvement and 17 (47%) items showed a deterioration. Post-TRIPOD, there was no improved reporting for articles citing the TRIPOD statement, nor for articles published in journals that published the TRIPOD statement. The methodological quality improved in the post-TRIPOD period. More models were externally validated in the same article (absolute difference 8%, post-TRIPOD: 39%), used measures of calibration (21%, post-TRIPOD: 87%) and discrimination (9%, post-TRIPOD: 100%), and used multiple imputation for handling missing data (12%, post-TRIPOD: 50%). Conclusions Since the publication of the TRIPOD statement, some reporting and methodological aspects have improved. Prediction models are still often poorly developed and validated and many aspects remain poorly reported, hindering optimal clinical application of these models. Long-term effects of the TRIPOD statement publication should be evaluated in future studies. Show less
Jenniskens, K.; Naaktgeboren, C.A.; Reitsma, J.B.; Hooft, L.; Moons, K.G.M.; Smeden, M. van 2019
Objectives: The objective of this study was to study the impact of ignoring uncertainty by forcing dichotomous classification (presence or absence) of the target disease on estimates of diagnostic... Show moreObjectives: The objective of this study was to study the impact of ignoring uncertainty by forcing dichotomous classification (presence or absence) of the target disease on estimates of diagnostic accuracy of an index test.Study Design and Setting: We evaluated the bias in estimated index test accuracy when forcing an expert panel to make a dichotomous target disease classification for each individual. Data for various scenarios with expert panels were simulated by varying the number and accuracy of "component reference tests" available to the expert panel, index test sensitivity and specificity, and target disease prevalence.Results: Index test accuracy estimates are likely to be biased when there is uncertainty surrounding the presence or absence of the target disease. Direction and amount of bias depend on the number and accuracy of component reference tests, target disease prevalence, and the true values of index test sensitivity and specificity.Conclusion: In this simulation, forcing expert panels to make a dichotomous decision on target disease classification in the presence of uncertainty leads to biased estimates of index test accuracy. Empirical studies are needed to demonstrate whether this bias can be reduced by assigning a probability of target disease presence for each individual, or using advanced statistical methods to account for uncertainty in target disease classification. (C) 2019 Elsevier Inc. All rights reserved. Show less
