Calibration of prothrombin time (PT) in terms of international normalized ratio (INR) has been outlined in “Guidelines for thromboplastins and plasmas used to control oral anticoagulant therapy” ... Show moreCalibration of prothrombin time (PT) in terms of international normalized ratio (INR) has been outlined in “Guidelines for thromboplastins and plasmas used to control oral anticoagulant therapy” (World Health Organization, 2013). The international standard ISO 17511:2020 presents requirements for manufacturers of in vitro diagnostic (IVD) medical devices (MDs) for documenting the calibration hierarchy for a measured quantity in human samples using a specified IVD MD. The objective of this article is to define an unequivocal, metrologically traceable calibration hierarchy for the INR measured in plasma as well as in whole blood samples. Calibration of PT and INR for IVD MDs according to World Health Organization guidelines is similar to that in cases where there is a reference measurement procedure that defines the measurand for value assignment as described in ISO 17511:2020. We conclude that, for PT/INR standardization, the optimal calibration hierarchy includes a primary process to prepare an international reference reagent and measurement procedure that defines the measurand by a value assignment protocol conforming to clause 5.3 of ISO 17511:2020. A panel of freshly prepared human plasma samples from healthy adult individuals and patients on vitamin K antagonists is used as a commutable secondary calibrator as described in ISO 17511:2020. A sustainable metrologically traceable calibration hierarchy for INR should be based on an international protocol for value assignment with a single primary reference thromboplastin and the harmonized manual tilt tube technique for clotting time determination. The primary international reference thromboplastin reagent should be used only for calibration of successive batches of the secondary reference thromboplastin reagent Show less
Ramspek, C.L.; Teece, L.; Snell, K.I.E.; Evans, M.; Riley, R.D.; Smeden, M. van; ... ; Diepen, M. van 2021
Background: External validation of prognostic models is necessary to assess the accuracy and generalizability of the model to new patients. If models are validated in a setting in which competing... Show moreBackground: External validation of prognostic models is necessary to assess the accuracy and generalizability of the model to new patients. If models are validated in a setting in which competing events occur, these competing risks should be accounted for when comparing predicted risks to observed outcomes. Methods: We discuss existing measures of calibration and discrimination that incorporate competing events for time-to-event models. These methods are illustrated using a clinical-data example concerning the prediction of kidney failure in a population with advanced chronic kidney disease (CKD), using the guideline-recommended Kidney Failure Risk Equation (KFRE). The KFRE was developed using Cox regression in a diverse population of CKD patients and has been proposed for use in patients with advanced CKD in whom death is a frequent competing event. Results: When validating the 5-year KFRE with methods that account for competing events, it becomes apparent that the 5-year KFRE considerably overestimates the real-world risk of kidney failure. The absolute overestimation was 10%age points on average and 29%age points in older high-risk patients. Conclusions: It is crucial that competing events are accounted for during external validation to provide a more reliable assessment the performance of a model in clinical settings in which competing risks occur. Show less
Jurgens-Lahnstein, J.H.; Petersen, E.T.; Laursen, M.; Iversen, C.H.; Kaptein, B.L.; Lindgren, L.; Stilling, M. 2021
Radiostereometric analysis (RSA) is an accurate and precise radiographic method that can be used to measure micromotion of implants and study joint kinematics in vivo. A calibration cage with... Show moreRadiostereometric analysis (RSA) is an accurate and precise radiographic method that can be used to measure micromotion of implants and study joint kinematics in vivo. A calibration cage with radiopaque markers is used to calibrate the RSA images; however, the thickness (250 mm) of the calibration cage restricts the available area for the patient and equipment during RSA recordings. A thinner calibration cage would increase the recording area, facilitate handling of the cage, and ease integration of the cage with the RSA system. We developed a thinner calibration cage without compromise of accuracy and precision. First, we performed numerical simulations of an RSA system, and showed that the calibration cage thickness could be decreased to 140 mm maintaining accuracy and precision using 40 fiducial and 30 control markers. Second, we constructed a new calibration cage (NRT cage) according to the simulation results. Third, we validated the new calibration cage against two state-of-the-art calibration cages (Umeaa cage and Leiden cage) in a phantom study. All cages performed similar for marker-based analysis, except for y-rotation, where the Umeaa cage (SD = 0.064 mm) was less precise compared to the NRT (SD = 0.038 mm) and Leiden cages (0.042 mm) (p = .01). For model-based analysis the NRT cage had superior precision for translations (SD <= 0.054 mm) over the Leiden cage (SD <= 0.118 mm) and Umeaa cage (SD <= 0.093 mm) (p < .01). The combined study confirmed that the new and thinner calibration cage maintained accuracy and precision at the level of existing thicker calibration cages. Show less
Background: There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally... Show moreBackground: There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated. Methods: A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data. Results: We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0-94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th-75th percentile [interquartile range (IQR)], 0.66-0.79), representing a median percent decrease in discrimination of -11.1% (IQR, -32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was -3.7% (IQR, -13.2 to 3.1) for closely related validations (n=123), -9.0 (IQR, -27.6 to 3.9) for related validations (n=862), and -17.2% (IQR, -42.3 to 0) for distantly related validations (n=717; P<0.001). Conclusions: Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings. Show less
Pavlou, M.; Qu, C.; Omar, R.Z.; Seaman, S.R.; Steyerberg, E.W.; White, I.R.; Ambler, G. 2021
Risk-prediction models for health outcomes are used in practice as part of clinical decision-making, and it is essential that their performance be externally validated. An important aspect in the... Show moreRisk-prediction models for health outcomes are used in practice as part of clinical decision-making, and it is essential that their performance be externally validated. An important aspect in the design of a validation study is choosing an adequate sample size. In this paper, we investigate the sample size requirements for validation studies with binary outcomes to estimate measures of predictive performance (C-statistic for discrimination and calibration slope and calibration in the large). We aim for sufficient precision in the estimated measures. In addition, we investigate the sample size to achieve sufficient power to detect a difference from a target value. Under normality assumptions on the distribution of the linear predictor, we obtain simple estimators for sample size calculations based on the measures above. Simulation studies show that the estimators perform well for common values of the C-statistic and outcome prevalence when the linear predictor is marginally Normal. Their performance deteriorates only slightly when the normality assumptions are violated. We also propose estimators which do not require normality assumptions but require specification of the marginal distribution of the linear predictor and require the use of numerical integration. These estimators were also seen to perform very well under marginal normality. Our sample size equations require a specified standard error (SE) and the anticipated C-statistic and outcome prevalence. The sample size requirement varies according to the prognostic strength of the model, outcome prevalence, choice of the performance measure and study objective. For example, to achieve an SE < 0.025 for the C-statistic, 60-170 events are required if the true C-statistic and outcome prevalence are between 0.64-0.85 and 0.05-0.3, respectively. For the calibration slope and calibration in the large, achieving SE < 0.15 would require 40-280 and 50-100 events, respectively. Our estimators may also be used for survival outcomes when the proportion of censored observations is high. Show less
Background:Arrhythmogenic right ventricular cardiomyopathy (ARVC) is associated with ventricular arrhythmias (VA) and sudden cardiac death (SCD). A model was recently developed to predict incident... Show moreBackground:Arrhythmogenic right ventricular cardiomyopathy (ARVC) is associated with ventricular arrhythmias (VA) and sudden cardiac death (SCD). A model was recently developed to predict incident sustained VA in patients with ARVC. However, since this outcome may overestimate the risk for SCD, we aimed to specifically predict life-threatening VA (LTVA) as a closer surrogate for SCD.Methods:We assembled a retrospective cohort of definite ARVC cases from 15 centers in North America and Europe. Association of 8 prespecified clinical predictors with LTVA (SCD, aborted SCD, sustained, or implantable cardioverter-defibrillator treated ventricular tachycardia >250 beats per minute) in follow-up was assessed by Cox regression with backward selection. Candidate variables included age, sex, prior sustained VA (>= 30s, hemodynamically unstable, or implantable cardioverter-defibrillator treated ventricular tachycardia; or aborted SCD), syncope, 24-hour premature ventricular complexes count, the number of anterior and inferior leads with T-wave inversion, left and right ventricular ejection fraction. The resulting model was internally validated using bootstrapping.Results:A total of 864 patients with definite ARVC (40 +/- 16 years; 53% male) were included. Over 5.75 years (interquartile range, 2.77-10.58) of follow-up, 93 (10.8%) patients experienced LTVA including 15 with SCD/aborted SCD (1.7%). Of the 8 prespecified clinical predictors, only 4 (younger age, male sex, premature ventricular complex count, and number of leads with T-wave inversion) were associated with LTVA. Notably, prior sustained VA did not predict subsequent LTVA (P=0.850). A model including only these 4 predictors had an optimism-corrected C-index of 0.74 (95% CI, 0.69-0.80) and calibration slope of 0.95 (95% CI, 0.94-0.98) indicating minimal over-optimism.Conclusions:LTVA events in patients with ARVC can be predicted by a novel simple prediction model using only 4 clinical predictors. Prior sustained VA and the extent of functional heart disease are not associated with subsequent LTVA events. Show less
Mertens, B.J.A.; Banzato, E.; Wreede, L.C. de 2020
We investigate calibration and assessment of predictive rules when missing values are present in the predictors. Our paper has two key objectives. The first is to investigate how the calibration of... Show moreWe investigate calibration and assessment of predictive rules when missing values are present in the predictors. Our paper has two key objectives. The first is to investigate how the calibration of the prediction rule can be combined with use of multiple imputation to account for missing predictor observations. The second objective is to propose such methods that can be implemented with current multiple imputation software, while allowing for unbiased predictive assessment through validation on new observations for which outcome is not yet available.We commence with a review of the methodological foundations of multiple imputation as a model estimation approach as opposed to a purely algorithmic description. We specifically contrast application of multiple imputation for parameter (effect) estimation with predictive calibration. Based on this review, two approaches are formulated, of which the second utilizes application of the classical Rubin's rules for parameter estimation, while the first approach averages probabilities from models fitted on single imputations to directly approximate the predictive density for future observations. We present implementations using current software that allow for validation and estimation of performance measures by cross-validation, as well as imputation of missing data in predictors on the future data where outcome is missing by definition.To simplify, we restrict discussion to binary outcome and logistic regression throughout. Method performance is verified through application on two real data sets. Accuracy (Brier score) and variance of predicted probabilities are investigated. Results show substantial reductions in variation of calibrated probabilities when using the first approach. Show less
Uveal melanoma (UM) is fatal in -50% of patients as a result of disseminated disease. This study aims to externally validate the Liverpool Uveal Melanoma Prognosticator Online V3 (LUMPO3) to... Show moreUveal melanoma (UM) is fatal in -50% of patients as a result of disseminated disease. This study aims to externally validate the Liverpool Uveal Melanoma Prognosticator Online V3 (LUMPO3) to determine its reliability in predicting survival after treatment for choroidal melanoma when utilizing external data from other ocular oncology centers. Anonymized data of 1836 UM patients from seven international ocular oncology centers were analyzed with LUMPO3 to predict the 10-year survival for each patient in each external dataset. The analysts were masked to the patient outcomes. Model predictions were sent to an independent statistician to evaluate LUMPO3's performance using discrimination and calibration methods. LUMPO3's ability to discriminate between UM patients who died of metastatic UM and those who were still alive was fair-to-good, with C-statistics ranging from 0.64 to 0.85 at year 1. The pooled estimate for all external centers was 0.72 (95% confidence interval: 0.68 to 0.75). Agreement between observed and predicted survival probabilities was generally good given differences in case mix and survival rates between different centers. Despite the differences between the international cohorts of patients with primary UM, LUMPO3 is a valuable tool for predicting all-cause mortality in this disease when using data from external centers. Show less
