The influx of technology in education has made it increasingly difficult to assess the validity of educational assessments. The field of information systems often ignores the social dimension... Show moreThe influx of technology in education has made it increasingly difficult to assess the validity of educational assessments. The field of information systems often ignores the social dimension during validation, whereas educational research neglects the technical dimensions of designed instruments. The inseparability of social and technical elements forms the bedrock of socio-technical systems. Therefore, the current lack of validation approaches that address both dimensions is a significant gap. We address this gap by introducing VAST: a validation framework for e-assessment solutions. Examples of such solutions are technology-enhanced learning systems and e-health applications. Using multi-grounded action research as our methodology, we investigate how we can synthesise existing knowledge from information systems and educational measurement to construct our validation framework. We develop an extensive user guideline complementing our framework and find through expert interviews that VAST facilitates a comprehensive, practical approach to validating e-assessment solutions. Show less
Toledo, C. van; Schraagen, M.; Dijk, F. van; Brinkhuis, M.; Spruit, M. 2023
This paper introduces a novel method to predict when a Google translation is better than other machine translations (MT) in Dutch. Instead of considering fidelity, this approach considers fluency... Show moreThis paper introduces a novel method to predict when a Google translation is better than other machine translations (MT) in Dutch. Instead of considering fidelity, this approach considers fluency and readability indicators for when Google ranked best. This research explores an alternative approach in the field of quality estimation. The paper contributes by publishing a dataset with sentences from English to Dutch, with human-made classifications on a best-worst scale. Logistic regression shows a correlation between T-Scan output, such as readability measurements like lemma frequencies, and when Google translation was better than Azure and IBM. The last part of the results section shows the prediction possibilities. First by logistic regression and second by a generated automated machine learning model. Respectively, they have an accuracy of 0.59 and 0.61. Show less
Background Endometrial cancer can be molecularly classified into POLEmut , mismatch repair deficient (MMRd), p53 abnormal (p53abn), and no specific molecular profile (NSMP) subgroups. We aimed to... Show moreBackground Endometrial cancer can be molecularly classified into POLEmut , mismatch repair deficient (MMRd), p53 abnormal (p53abn), and no specific molecular profile (NSMP) subgroups. We aimed to develop an interpretable deep learning pipeline for whole-slide-image-based prediction of the four molecular classes in endometrial cancer (im4MEC), to identify morpho-molecular correlates, and to refine prognostication. Methods This combined analysis included diagnostic haematoxylin and eosin-stained slides and molecular and clinicopathological data from 2028 patients with intermediate-to-high-risk endometrial cancer from the PORTEC-1 (n=466), PORTEC-2 (n=375), and PORTEC-3 (n=393) randomised trials and the TransPORTEC pilot study (n=110), the Medisch Spectrum Twente cohort (n=242), a case series of patients with POLEmut endometrial cancer in the Leiden Endometrial Cancer Repository (n=47), and The Cancer Genome Atlas-Uterine Corpus Endometrial Carcinoma cohort (n=395). PORTEC-3 was held out as an independent test set and a four-fold cross validation was performed. Performance was measured with the macro and class-wise area under the receiver operating characteristic curve (AUROC). Whole-slide images were segmented into tiles of 360 & mu;m resized to 224 x 224 pixels. im4MEC was trained to learn tile-level morphological features with self-supervised learning and to molecularly classify whole-slide images with an attention mechanism. The top 20 tiles with the highest attention scores were reviewed to identify morpho-molecular correlates. Predictions of a nuclear classification deep learning model serve to derive interpretable morphological features. We analysed 5-year recurrence-free survival and explored prognostic refinement by molecular class using the Kaplan-Meier method. Findings im4MEC attained macro-average AUROCs of 0 & BULL;874 (95% CI 0 & BULL;856-0 & BULL;893) on four-fold cross-validation and 0 & BULL;876 on the independent test set. The class-wise AUROCs were 0 & BULL;849 for POLEmut (n=51), 0 & BULL;844 for MMRd (n=134), 0 & BULL;883 for NSMP (n=120), and 0 & BULL;928 for p53abn (n=88). POLEmut and MMRd tiles had a high density of lymphocytes, p53abn tiles had strong nuclear atypia, and the morphology of POLEmut and MMRd endometrial cancer overlapped. im4MEC highlighted a low tumour-to-stroma ratio as a potentially novel characteristic feature of the NSMP class. 5-year recurrence-free survival was significantly different between im4MEC predicted molecular classes in PORTEC-3 (log-rank p<0 & BULL;0001). The ten patients with aggressive p53abn endometrial cancer that was predicted as MMRd showed inflammatory morphology and appeared to have a better prognosis than patients with correctly predicted p53abn endometrial cancer (p=0 & BULL;30). The four patients with NSMP endometrial cancer that was predicted as p53abn showed higher nuclear atypia and appeared to have a worse prognosis than patients with correctly predicted NSMP (p=0 & BULL;13). Patients with MMRd endometrial cancer predicted as POLEmut had an excellent prognosis, as do those with true POLEmut endometrial cancer. Interpretation We present the first interpretable deep learning model, im4MEC, for haematoxylin and eosin-based prediction of molecular endometrial cancer classification. im4MEC robustly identified morpho-molecular correlates and could enable further prognostic refinement of patients with endometrial cancer. Copyright & COPY; 2022 The Author(s). Published by Elsevier Ltd. This is an Open Access article under the CC BY 4.0 license. Show less
Background: Risk-assessment of endometrial cancer (EC) is based on clinicopathological factors and molecular subgroup. It is unclear whether adding hormone receptor expression, L1CAM expression or... Show moreBackground: Risk-assessment of endometrial cancer (EC) is based on clinicopathological factors and molecular subgroup. It is unclear whether adding hormone receptor expression, L1CAM expression or CTNNB1 status yields prognostic refinement. Methods: Paraffin-embedded tumour samples of women with high-risk EC (HR-EC) from the PORTEC-3 trial (n = 424), and a Dutch prospective clinical cohort called MST (n = 256), were used. All cases were molecularly classified. Expression of L1CAM, ER and PR were analysed by whole-slide immunohistochemistry and CTNNB1 mutations were assessed with a next-generation sequencing. Kaplan-Meier method, log-rank tests and Cox's proportional hazard models were used for survival analysis. Results: In total, 648 HR-EC were included. No independent prognostic value of ER, PR, L1CAM, and CTNNB1 was found, while age, stage, and adjuvant chemotherapy had an independent impact on risk of recurrence. Subgroup-analysis showed that only in NSMP HR-EC, ER-positivity was independently associated with a reduced risk of recurrence (HR 0.33, 95%CI 0.15-0.75). Conclusions: We confirmed the prognostic impact of the molecular classification, age, stage, and adjuvant CTRT in a large cohort of high-risk EC. ER-positivity is a strong favourable prognostic factor in NSMP HR-EC and identifies a homogeneous subgroup of NSMP tumours. Assessment of ER status in high-risk NSMP EC is feasible in clinical practice and could improve risk stratification and treatment. Show less
Toledo, C. van; Schraagen, M.; Dijk, F. van; Brinkhuis, M.; Spruit, M. 2022
We explore the use case of question answering (QA) by a contact centre for 130,000 Dutch government employees in the domain of questions about human resources (HR). HR questions can be answered... Show moreWe explore the use case of question answering (QA) by a contact centre for 130,000 Dutch government employees in the domain of questions about human resources (HR). HR questions can be answered using personnel files or general documentation, with the latter being the focus of the current research. We created a Dutch HR QA dataset with over 300 questions in the format of the Squad 2.0 dataset, which distinguishes between answerable and unanswerable questions. We applied various BERT-based models, either directly or after finetuning on the new dataset. The F1-scores reached 0.47 for unanswerable questions and 1.0 for answerable questions depending on the topic; however, large variations in scores were observed. We conclude more data are needed to further improve the performance of this task. Show less
Haastrecht, M. van; Golpur, G.; Tzismadia, G.; Kab, R.; Priboi, C.; David, D.; ... ; Spruit, M. 2022
Small- and medium-sized enterprises (SMEs) frequently experience cyberattacks, but often do not have the means to counter these attacks. Therefore, cybersecurity researchers and practitioners need... Show moreSmall- and medium-sized enterprises (SMEs) frequently experience cyberattacks, but often do not have the means to counter these attacks. Therefore, cybersecurity researchers and practitioners need to aid SMEs in their defence against cyber threats. Research has shown that SMEs require solutions that are automated and adapted to their context. In recent years, we have seen a surge in initiatives to share cyber threat intelligence (CTI) to improve collective cybersecurity resilience. Shared CTI has the potential to answer the SME call for automated and adaptable solutions. Sadly, as we demonstrate in this paper, current shared intelligence approaches scarcely address SME needs. We must investigate how shared CTI can be used to improve SME cybersecurity resilience. In this paper, we tackle this challenge using a systematic review to discover current state-of-the-art approaches to using shared CTI. We find that threat intelligence sharing platforms such as MISP have the potential to address SME needs, provided that the shared intelligence is turned into actionable insights. Based on this observation, we developed a prototype application that processes MISP data automatically, prioritises cybersecurity threats for SMEs, and provides SMEs with actionable recommendations tailored to their context. Subsequent evaluations in operational environments will help to improve our application, such that SMEs are enabled to thwart cyberattacks in future. Show less
Voets, M.M.; Groothuis-Oudshoorn, C.G.M.; Veneklaas, L.H.J.; Manohar, S.; Brinkhuis, M.; Veltman, J.; ... ; Siesling, S. 2021
The goal of this study was to describe the variation in hospital-based diagnostic care activities for patients with symptomatology suspect for breast cancer in The Netherlands. Two cohorts were... Show moreThe goal of this study was to describe the variation in hospital-based diagnostic care activities for patients with symptomatology suspect for breast cancer in The Netherlands. Two cohorts were included: the 'benign' cohort (30,334 women suspected of, but without breast cancer) and the 'malignant' cohort (2236 breast cancer patients). Hospital-based financial data was combined with tumor data (malignant cohort) from The Netherlands Cancer Registry. Patterns within diagnostic pathways were analyzed. Factors influencing the number of visits and number of diagnostic care activities until diagnosis were identified in the malignant cohort with multivariable Cox and Poisson regression models. Compared to patients with benign diagnosis, patients with malignant disease received their diagnosis less frequently in one day, after an equal average number of hospital visits and higher average number of diagnostic activities. Factors increasing the number of diagnostic care activities were the following: lower age and higher cM-and cN-stages. Factors increasing the number of days until (malignant) diagnosis were as follows: higher BIRADS-score, screen-detected and higher cN-and cT-stages. Hospital of diagnosis influenced both number of activities and days to diagnosis. The diagnostic care pathway of patients with malignant disease required more time and diagnostic activities than benign disease and depends on hospital, tumor and patient characteristics. Show less
Small- and medium-sized enterprises (SMEs) frequently experience cyberattacks, but often do not have the means to counter these attacks. Therefore, cybersecurity researchers and practitioners need... Show moreSmall- and medium-sized enterprises (SMEs) frequently experience cyberattacks, but often do not have the means to counter these attacks. Therefore, cybersecurity researchers and practitioners need to aid SMEs in their defence against cyber threats. Research has shown that SMEs require solutions that are automated and adapted to their context. In recent years, we have seen a surge in initiatives to share cyber threat intelligence (CTI) to improve collective cybersecurity resilience. Shared CTI has the potential to answer the SME call for automated and adaptable solutions. Sadly, as we demonstrate in this paper, current shared intelligence approaches scarcely address SME needs. We must investigate how shared CTI can be used to improve SME cybersecurity resilience. In this paper, we tackle this challenge using a systematic review to discover current state-of-the-art approaches to using shared CTI. We find that threat intelligence sharing platforms such as MISP have the potential to address SME needs, provided that the shared intelligence is turned into actionable insights. Based on this observation, we developed a prototype application that processes MISP data automatically, prioritises cybersecurity threats for SMEs, and provides SMEs with actionable recommendations tailored to their context. Subsequent evaluations in operational environments will help to improve our application, such that SMEs are enabled to thwart cyberattacks in future. Show less
Haastrecht, M. van; Ozkan, B.Y.; Brinkhuis, M.; Spruit, M. 2021
Featured Application The results of this work will be incorporated in an application for SMEs in Europe, which aims to improve cybersecurity awareness and resilience, as part of the EU Horizon 2020... Show moreFeatured Application The results of this work will be incorporated in an application for SMEs in Europe, which aims to improve cybersecurity awareness and resilience, as part of the EU Horizon 2020 GEIGER project. Cybersecurity threats are on the rise, and small- and medium-sized enterprises (SMEs) struggle to cope with these developments. To combat threats, SMEs must first be willing and able to assess their cybersecurity posture. Cybersecurity risk assessment, generally performed with the help of metrics, provides the basis for an adequate defense. Significant challenges remain, however, especially in the complex socio-technical setting of SMEs. Seemingly basic questions, such as how to aggregate metrics and ensure solution adaptability, are still open to debate. Aggregation and adaptability are vital topics to SMEs, as they require the assimilation of metrics into an actionable advice adapted to their situation and needs. To address these issues, we systematically review socio-technical cybersecurity metric research in this paper. We analyse aggregation and adaptability considerations and investigate how current findings apply to the SME situation. To ensure that we provide valuable insights to researchers and practitioners, we integrate our results in a novel socio-technical cybersecurity framework geared towards the needs of SMEs. Our framework allowed us to determine a glaring need for intuitive, threat-based cybersecurity risk assessment approaches for the least digitally mature SMEs. In the future, we hope our framework will help to offer SMEs some deserved respite by guiding the design of suitable cybersecurity assessment solutions. Show less
Haastrecht, M. van; Ozkan, B.Y.; Brinkhuis, M.; Spruit, M. 2021
Research output has grown significantly in recent years, often making it difficult to see the forest for the trees. Systematic reviews are the natural scientific tool to provide clarity in these... Show moreResearch output has grown significantly in recent years, often making it difficult to see the forest for the trees. Systematic reviews are the natural scientific tool to provide clarity in these situations. However, they are protracted processes that require expertise to execute. These are problematic characteristics in a constantly changing environment. To solve these challenges, we introduce an innovative systematic review methodology: SYMBALS. SYMBALS blends the traditional method of backward snowballing with the machine learning method of active learning. We applied our methodology in a case study, demonstrating its ability to swiftly yield broad research coverage. We proved the validity of our method using a replication study, where SYMBALS was shown to accelerate title and abstract screening by a factor of 6. Additionally, four benchmarking experiments demonstrated the ability of our methodology to outperform the state-of-the-art systematic review methodology FAST2. Show less
Background Three tools are currently available to predict the risk of contralateral breast cancer (CBC). We aimed to compare the performance of the Manchester formula, CBCrisk, and PredictCBC in... Show moreBackground Three tools are currently available to predict the risk of contralateral breast cancer (CBC). We aimed to compare the performance of the Manchester formula, CBCrisk, and PredictCBC in patients with invasive breast cancer (BC). Methods We analyzed data of 132,756 patients (4682 CBC) from 20 international studies with a median follow-up of 8.8 years. Prediction performance included discrimination, quantified as a time-dependent Area-Under-the-Curve (AUC) at 5 and 10 years after diagnosis of primary BC, and calibration, quantified as the expected-observed (E/O) ratio at 5 and 10 years and the calibration slope. Results The AUC at 10 years was: 0.58 (95% confidence intervals [CI] 0.57-0.59) for CBCrisk; 0.60 (95% CI 0.59-0.61) for the Manchester formula; 0.63 (95% CI 0.59-0.66) and 0.59 (95% CI 0.56-0.62) for PredictCBC-1A (for settings where BRCA1/2 mutation status is available) and PredictCBC-1B (for the general population), respectively. The E/O at 10 years: 0.82 (95% CI 0.51-1.32) for CBCrisk; 1.53 (95% CI 0.63-3.73) for the Manchester formula; 1.28 (95% CI 0.63-2.58) for PredictCBC-1A and 1.35 (95% CI 0.65-2.77) for PredictCBC-1B. The calibration slope was 1.26 (95% CI 1.01-1.50) for CBCrisk; 0.90 (95% CI 0.79-1.02) for PredictCBC-1A; 0.81 (95% CI 0.63-0.99) for PredictCBC-1B, and 0.39 (95% CI 0.34-0.43) for the Manchester formula. Conclusions Current CBC risk prediction tools provide only moderate discrimination and the Manchester formula was poorly calibrated. Better predictors and re-calibration are needed to improve CBC prediction and to identify low- and high-CBC risk patients for clinical decision-making. Show less
Background: Breast cancer survivors are at risk for contralateral breast cancer (CBC), with the consequent burden of further treatment and potentially less favorable prognosis. We aimed to develop... Show moreBackground: Breast cancer survivors are at risk for contralateral breast cancer (CBC), with the consequent burden of further treatment and potentially less favorable prognosis. We aimed to develop and validate a CBC risk prediction model and evaluate its applicability for clinical decision-making.Methods: We included data of 132,756 invasive non-metastatic breast cancer patients from 20 studies with 4682 CBC events and a median follow-up of 8.8 years. We developed a multivariable Fine and Gray prediction model (PredictCBC-1A) including patient, primary tumor, and treatment characteristics and BRCA1/2 germline mutation status, accounting for the competing risks of death and distant metastasis. We also developed a model without BRCA1/2 mutation status (PredictCBC-1B) since this information was available for only 6% of patients and is routinely unavailable in the general breast cancer population. Prediction performance was evaluated using calibration and discrimination, calculated by a time-dependent area under the curve (AUC) at 5 and 10 years after diagnosis of primary breast cancer, and an internal-external cross-validation procedure. Decision curve analysis was performed to evaluate the net benefit of the model to quantify clinical utility.Results: In the multivariable model, BRCA1/2 germline mutation status, family history, and systemic adjuvant treatment showed the strongest associations with CBC risk. The AUC of PredictCBC-1A was 0.63 (95% prediction interval (PI) at 5 years, 0.52-0.74; at 10 years, 0.53-0.72). Calibration-in-the-large was -0.13 (95% PI: -1.62-1.37), and the calibration slope was 0.90 (95% PI: 0.73-1.08). The AUC of Predict-1B at 10 years was 0.59 (95% PI: 0.52-0.66); calibration was slightly lower. Decision curve analysis for preventive contralateral mastectomy showed potential clinical utility of PredictCBC-1A between thresholds of 4-10% 10-year CBC risk for BRCA1/2 mutation carriers and non-carriers.Conclusions: We developed a reasonably calibrated model to predict the risk of CBC in women of European-descent; however, prediction accuracy was moderate. Our model shows potential for improved risk counseling, but decision-making regarding contralateral preventive mastectomy, especially in the general breast cancer population where limited information of the mutation status in BRCA1/2 is available, remains challenging. Show less
Schmidt, M.K.; Broek, A.J. van den; Tollenaar, R.A.E.M.; Smit, V.T.H.B.M.; Westenend, P.J.; Brinkhuis, M.; ... ; Veer, L.J. van 't 2017
