To promote cross-community dialogue on matters of significance within the field of learning analytics (LA), we as editors-in-chief of the Journal of Learning Analytics (JLA) have introduced a... Show moreTo promote cross-community dialogue on matters of significance within the field of learning analytics (LA), we as editors-in-chief of the Journal of Learning Analytics (JLA) have introduced a section for papers that are open to peer commentary. An invitation to submit proposals for commentaries on the paper was released, and 12 of these proposals were accepted. The 26 authors of the accepted commentaries are based in Europe, North America, and Australia. They range in experience from PhD students and early-career researchers to some of the longest-standing, most senior members of the learning analytics community. This paper brings those commentaries together, and we recommend reading it as a companion piece to the original paper by Motz et al. (2023), which also appears in this issue. Show less
The influx of technology in education has made it increasingly difficult to assess the validity of educational assessments. The field of information systems often ignores the social dimension... Show moreThe influx of technology in education has made it increasingly difficult to assess the validity of educational assessments. The field of information systems often ignores the social dimension during validation, whereas educational research neglects the technical dimensions of designed instruments. The inseparability of social and technical elements forms the bedrock of socio-technical systems. Therefore, the current lack of validation approaches that address both dimensions is a significant gap. We address this gap by introducing VAST: a validation framework for e-assessment solutions. Examples of such solutions are technology-enhanced learning systems and e-health applications. Using multi-grounded action research as our methodology, we investigate how we can synthesise existing knowledge from information systems and educational measurement to construct our validation framework. We develop an extensive user guideline complementing our framework and find through expert interviews that VAST facilitates a comprehensive, practical approach to validating e-assessment solutions. Show less
Dijk, B.M.A. van; Spruit, M.R.; Duijn, M.J. van 2023
Children are the focal point for studying the link between language and Theory of Mind (ToM) competence. Language and ToM are often studied with younger children and standardized tests, but as both... Show moreChildren are the focal point for studying the link between language and Theory of Mind (ToM) competence. Language and ToM are often studied with younger children and standardized tests, but as both are social competences, data and methods with higher ecological validity are critical.We leverage a corpus of 442 freely-told stories by Dutch children aged 4-12, recorded in their everyday classroom environments, to study language and ToM with NLP-tools. We labelled stories according to the mental depth of story characters children create, as a proxy for their ToM competence ‘in action’, and built a classifier with features encoding linguistic competences identified in existing work as predictive of ToM.We obtain good and fairly robust results (F1-macro = .71), relative to the complexity of the task for humans. Our results are explainable in that we link specific linguistic features such as lexical complexity and sentential complementation, that are relatively independent of children’s ages, to higher levels of character depth. This confirms and extends earlier work, as our study includes older children and socially embedded data from a different domain. Overall, our results support the idea that language and ToM are strongly interlinked, and that in narratives the former can scaffold the latter. Show less
Inaugural lecture by Prof.dr. Marco Spruit on the acceptance of the position of professor of Advanced Data Science in Population Health at Leiden University on 1 April 2022
Background The Screening Tool of Older Persons' Prescriptions (STOPP)/Screening Tool to Alert to Right Treatment (START) instrument is used to evaluate the appropriateness of medication in older... Show moreBackground The Screening Tool of Older Persons' Prescriptions (STOPP)/Screening Tool to Alert to Right Treatment (START) instrument is used to evaluate the appropriateness of medication in older people. STOPP/START criteria have been converted into software algorithms and implemented in a clinical decision support system (CDSS) to facilitate their use in clinical practice. Objective Our objective was to determine the frequency of CDSS-generated STOPP/START signals and their subsequent acceptance by a pharmacotherapy team in a hospital setting. Design and Methods Hospitalised older patients with polypharmacy and multimorbidity allocated to the intervention arm of the OPERAM (OPtimising thERapy to prevent Avoidable hospital admissions in the Multimorbid elderly) trial underwent a CDSS-assisted structured medication review in four European hospitals. We evaluated the frequency of CDSS-generated STOPP/START signals and the subsequent acceptance of these signals by a trained pharmacotherapy team consisting of a physician and pharmacist after evaluation of clinical applicability to the individual patient, prior to discussing pharmacotherapy optimisation recommendations with the patient and attending physicians. Multivariate linear regression analysis was used to investigate potential patient-related (e.g. age, number of co-morbidities and medications) and setting-related (e.g. ward type, country of inclusion) determinants for acceptance of STOPP and START signals. Results In 819/826 (99%) of the patients, at least one STOPP/START signal was generated using a set of 110 algorithms based on STOPP/START v2 criteria. Overall, 39% of the 5080 signals were accepted by the pharmacotherapy team. There was a high variability in the frequency and the subsequent acceptance of the individual STOPP/START criteria. The acceptance ranged from 2.5 to 75.8% for the top ten most frequently generated STOPP and START signals. The signal to stop a drug without a clinical indication was most frequently generated (28%), with more than half of the signals accepted (54%). No difference in mean acceptance of STOPP versus START signals was found. In multivariate analysis, most patient-related determinants did not predict acceptance, although the acceptance of START signals increased in patients with one or more hospital admissions (+ 7.9; 95% confidence interval [CI] 1.6-14.1) or one or more falls in the previous year (+ 7.1; 95% CI 0.7-13.4). A higher number of co-morbidities was associated with lower acceptance of STOPP (- 11.8%; 95% CI - 19.2 to - 4.5) and START (- 11.0%; 95% CI - 19.4 to - 2.6) signals for patients with more than nine and between seven and nine co-morbidities, respectively. For setting-related determinants, the acceptance differed significantly between the participating trial sites. Compared with Switzerland, the acceptance was higher in Ireland (STOPP: + 26.8%; 95% CI 16.8-36.7; START: + 31.1%; 95% CI 18.2-44.0) and in the Netherlands (STOPP: + 14.7%; 95% CI 7.8-21.7). Admission to a surgical ward was positively associated with acceptance of STOPP signals (+ 10.3%; 95% CI 3.8-16.8). Conclusion The involvement of an expert team in translating population-based CDSS signals to individual patients is essential, as more than half of the signals for potential overuse, underuse, and misuse were not deemed clinically appropriate in a hospital setting. Patient-related potential determinants were poor predictors of acceptance.Future research investigating factors that affect patients' and physicians' agreement with medication changes recommended by expert teams may provide further insight for implementation in clinical practice. Registration ClinicalTrials.gov Identifier: NCT02986425. Show less
Instant analysis of cybersecurity reports is a fundamental challenge for security experts as an immeasurable amount of cyber information is generated on a daily basis, which necessitates automated... Show moreInstant analysis of cybersecurity reports is a fundamental challenge for security experts as an immeasurable amount of cyber information is generated on a daily basis, which necessitates automated information extraction tools to facilitate querying and retrieval of data. Hence, we present Open-CyKG: an Open Cyber Threat Intelligence (CTI) Knowledge Graph (KG) framework that is constructed using an attention-based neural Open Information Extraction (OIE) model to extract valuable cyber threat information from unstructured Advanced Persistent Threat (APT) reports. More specifically, we first identify relevant entities by developing a neural cybersecurity Named Entity Recognizer (NER) that aids in labeling relation triples generated by the OIE model. Afterwards, the extracted structured data is canonicalized to build the KG by employing fusion techniques using word embeddings. As a result, security professionals can execute queries to retrieve valuable information from the Open-CyKG framework. Experimental results demonstrate that our proposed components that build up Open-CyKG outperform state-of-the-art models.1 (c) 2021 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). Show less
Small- and medium-sized enterprises (SMEs) frequently experience cyberattacks, but often do not have the means to counter these attacks. Therefore, cybersecurity researchers and practitioners need... Show moreSmall- and medium-sized enterprises (SMEs) frequently experience cyberattacks, but often do not have the means to counter these attacks. Therefore, cybersecurity researchers and practitioners need to aid SMEs in their defence against cyber threats. Research has shown that SMEs require solutions that are automated and adapted to their context. In recent years, we have seen a surge in initiatives to share cyber threat intelligence (CTI) to improve collective cybersecurity resilience. Shared CTI has the potential to answer the SME call for automated and adaptable solutions. Sadly, as we demonstrate in this paper, current shared intelligence approaches scarcely address SME needs. We must investigate how shared CTI can be used to improve SME cybersecurity resilience. In this paper, we tackle this challenge using a systematic review to discover current state-of-the-art approaches to using shared CTI. We find that threat intelligence sharing platforms such as MISP have the potential to address SME needs, provided that the shared intelligence is turned into actionable insights. Based on this observation, we developed a prototype application that processes MISP data automatically, prioritises cybersecurity threats for SMEs, and provides SMEs with actionable recommendations tailored to their context. Subsequent evaluations in operational environments will help to improve our application, such that SMEs are enabled to thwart cyberattacks in future. Show less
The Cross-Industry Standard Process for Data Mining (CRISP-DM), despite being the most popular data mining process for more than two decades, is known to leave those organizations lacking... Show moreThe Cross-Industry Standard Process for Data Mining (CRISP-DM), despite being the most popular data mining process for more than two decades, is known to leave those organizations lacking operational data mining experience puzzled and unable to start their data mining projects. This is especially apparent in the first phase of Business Understanding, at the conclusion of which, the data mining goals of the project at hand should be specified, which arguably requires at least a conceptual understanding of the knowledge discovery process. We propose to bridge this knowledge gap from a Data Science perspective by applying Natural Language Processing techniques (NLP) to the organizations' e-mail exchange repositories to extract explicitly stated business goals from the conversations, thus bootstrapping the Business Understanding phase of CRISP-DM. Our NLP-Automated Method for Business Understanding (NAMBU) generates a list of business goals which can subsequently be used for further specification of data mining goals. The validation of the results on the basis of comparison to the results of manual business goal extraction from the Enron corpus demonstrates the usefulness of our NAMBU method when applied to large datasets. Show less
Recently, the topic of research data management has appeared at the forefront of Open Science as a prerequisite for preserving and disseminating research data efficiently. At the same time,... Show moreRecently, the topic of research data management has appeared at the forefront of Open Science as a prerequisite for preserving and disseminating research data efficiently. At the same time, scientific laboratories still rely upon digital files that are processed by experimenters to analyze and communicate laboratory results. In this study, we first apply a forensic process to investigate the information quality of digital evidence underlying published results. Furthermore, we use semiotics to describe the quality of information recovered from storage systems with laboratory forensics techniques. Next, we formulate laboratory analytics capabilities based on the results of the forensics analysis. Laboratory forensics and analytics form the basis of research data management. Finally, we propose a conceptual overview of open science readiness, which combines laboratory forensics techniques and laboratory analytics capabilities to help overcome research data management challenges in the near future. Show less
Blum, M.R.; Sallevelt, B.T.G.M.; Spinewine, A.; O'Mahony, D.; Moutzouri, E.; Feller, M.; ... ; Rodondi, N. 2021
OBJECTIVETo examine the effect of optimising drug treatment on drug related hospital admissions in older adults with multimorbidity and polypharmacy admitted to hospital.DESIGNCluster randomised... Show moreOBJECTIVETo examine the effect of optimising drug treatment on drug related hospital admissions in older adults with multimorbidity and polypharmacy admitted to hospital.DESIGNCluster randomised controlled trial.SETTING110 clusters of inpatient wards within university based hospitals in four European countries (Switzerland, Netherlands, Belgium, and Republic of Ireland) defined by attending hospital doctors.PARTICIPANTS2008 older adults (>= 70 years) with multimorbidity (>= 3chronic conditions) and polypharmacy (>= 5 drugs used long term).INTERVENTIONClinical staff clusters were randomised to usual care or a structured pharmacotherapy optimisation intervention performed at the individual level jointly by a doctor and a pharmacist, with the support of a clinical decision software system deploying the screening tool of older person's prescriptions and screening tool to alert to the right treatment (STOPP/START) criteria to identify potentially inappropriate prescribing.MAIN OUTCOME MEASUREPrimary outcome was first drug related hospital admission within 12 months.RESULTS2008 older adults (median nine drugs) were randomised and enrolled in 54 intervention clusters (963 participants) and 56 control clusters (1045 participants) receiving usual care. In the intervention arm, 86.1% of participants (n=789) had inappropriate prescribing, with a mean of 2.75 (SD 2.24) STOPP/START recommendations for each participant. 62.2% (n=491) had >= 1 recommendation successfully implemented at two months, predominantly discontinuation of potentially inappropriate drugs. In the intervention group, 211 participants (21.9%) experienced a first drug related hospital admission compared with 234 (22.4%) in the control group. In the intention-to-treat analysis censored for death as competing event (n=375, 18.7%), the hazard ratio for first drug related hospital admission was 0.95 (95% confidence interval 0.77 to 1.17). In the per protocol analysis, the hazard ratio for a drug related hospital admission was 0.91 (0.69 to 1.19). The hazard ratio for first fall was 0.96 (0.79 to 1.15; 237 v263 first falls) and for death was 0.90 (0.71 to 1.13; 172 v 203 deaths).CONCLUSIONSInappropriate prescribing was common in older adults with multimorbidity and polypharmacy admitted to hospital and was reduced through an intervention to optimise pharmacotherapy, but without effect on drug related hospital admissions. Additional efforts are needed to identify pharmacotherapy optimisation interventions that reduce inappropriate prescribing and improve patient outcomes. Show less
Objectives Recruiting general practitioners (GPs) and their multimorbid older patients for trials is challenging for multiple reasons (e.g., high workload, limited mobility). The comparability of... Show moreObjectives Recruiting general practitioners (GPs) and their multimorbid older patients for trials is challenging for multiple reasons (e.g., high workload, limited mobility). The comparability of study participants is important for interpreting study findings. This manuscript describes the baseline characteristics of GPs and patients participating in the 'Optimizing PharmacoTherapy in older multimorbid adults In primary CAre' (OPTICA) trial, a study of optimization of pharmacotherapy for multimorbid older adults. The overall aim of this study was to determine if the GPs and patients participating in the OPTICA trial are comparable to the real-world population in Swiss primary care. Design Analysis of baseline data from GPs and patients in the OPTICA trial and a reference cohort from the FIRE ('Family medicine ICPC Research using Electronic medical records') project. Setting Primary care, Switzerland. Participants Three hundred twenty-three multimorbid (>= 3 chronic conditions) patients with polypharmacy (>= 5 regular medications) aged >= 65 years and 43 GPs recruited for the OPTICA trial were compared to 22,907 older multimorbid patients with polypharmacy and 227 GPs from the FIRE database. Methods We compared the characteristics of GPs and patients participating in the OPTICA trial with other GPs and other older multimorbid adults with polypharmacy in the FIRE database. We described the baseline willingness to have medications deprescribed of the patients participating in the OPTICA trial using the revised Patients' Attitudes Towards Deprescribing (rPATD) questionnaire. Results The GPs in the FIRE project and OPTICA were similar in terms of sociodemographic characteristics and their work as a GP (e.g. aged in their fifties, >= 10 years of experience, >= 60% are self-employed, >= 80% work in a group practice). The median age of patients in the OPTICA trial was 77 years and 45% of trial participants were women. Patients participating in the OPTICA trial and patients in the FIRE database were comparable in terms of age, certain clinical characteristics (e.g. systolic blood pressure, body mass index) and health services use (e.g. selected lab and vital data measurements). More than 80% of older multimorbid patients reported to be willing to stop >= 1 of their medications if their doctor said that this would be possible. Conclusion The characteristics of patients and GPs recruited into the OPTICA trial are relatively comparable to characteristics of a real-world Swiss population, which indicates that recruiting a generalizable patient sample is possible in the primary care setting. Multimorbid patients in the OPTICA trial reported a high willingness to have medications deprescribed. Show less
