Theoretical arguments and empirical investigations indicate that a high proportion of published findings do not replicate and are likely false. The current position paper provides a broad... Show moreTheoretical arguments and empirical investigations indicate that a high proportion of published findings do not replicate and are likely false. The current position paper provides a broad perspective on scientific error, which may lead to replication failures. This broad perspective focuses on reform history and on opportunities for future reform. We organize our perspective along four main themes: institutional reform, methodological reform, statistical reform and publishing reform. For each theme, we illustrate potential errors by narrating the story of a fictional researcher during the research cycle. We discuss future opportunities for reform. The resulting agenda provides a resource to usher in an era that is marked by a research culture that is less error-prone and a scientific publication landscape with fewer spurious findings. Show less
As the COVID-19 pandemic unfolds, researchers from all disciplines are coming together and contributing their expertise. CORD-19, a dataset of COVID-19 and coronavirus publications, has been made... Show moreAs the COVID-19 pandemic unfolds, researchers from all disciplines are coming together and contributing their expertise. CORD-19, a dataset of COVID-19 and coronavirus publications, has been made available alongside calls to help mine the information it contains and to create tools to search it more effectively. We analyse the delineation of the publications included in CORD-19 from a scientometric perspective. Based on a comparison to the Web of Science database, we find that CORD-19 provides an almost complete coverage of research on COVID-19 and coronaviruses. CORD-19 contains not only research that deals directly with COVID-19 and coronaviruses, but also research on viruses in general. Publications from CORD-19 focus mostly on a few well-defined research areas, in particular: coronaviruses (primarily SARS-CoV, MERS-CoV and SARS-CoV-2); public health and viral epidemics; molecular biology of viruses; influenza and other families of viruses; immunology and antivirals; clinical medicine. CORD-19 publications that appeared in 2020, especially editorials and letters, are disproportionately popular on social media. While we fully endorse the CORD-19 initiative, it is important to be aware that CORD-19 extends beyond research on COVID-19 and coronaviruses. Show less
Minority integration is a highly contested topic in public debates, and assimilationist actors appear to have gained discursive ground. However, it remains difficult to accurately depict how power... Show moreMinority integration is a highly contested topic in public debates, and assimilationist actors appear to have gained discursive ground. However, it remains difficult to accurately depict how power relations in debates change and evolve. In this study, the public debates on minority integration in Flanders and the Netherlands between 2006 and 2012 are studied to ascertain changing power relations. We use a relational method to identify clusters formed through discursive contention and study polarization in the debates as well as several aspects of discursive power between and within clusters. In the Netherlands, a pattern identified in earlier research is reproduced, whereby a unified but small cluster of assimilationists with strong discursive leaders is able to dominate the debate on integration. In Flanders, group consolidation is too low, so the clusters cannot be viewed as cohesive groups. Another difference to the Dutch debate is that the volume of opinion articles is much lower and the actors in the Flemish debate are more often foreign opinion leaders. We conclude that the assimilationists have increased their discursive power in the Dutch debate, while the anti-assimilationists have lost power. The stark contrast between the Dutch and Flemish discursive landscape highlights the need for more research on the causal mechanism behind discursive struggles. Show less
Akbaritabar, A.; Traag, V.A.; Caimo, A.; Squazzoni, F. 2020
Examining coauthorship networks is key to study scientific collaboration patterns and structural characteristics of scientific communities. Here, we studied coauthorship networks of sociologists in... Show moreExamining coauthorship networks is key to study scientific collaboration patterns and structural characteristics of scientific communities. Here, we studied coauthorship networks of sociologists in Italy, using temporal and multi-level quantitative analysis. By looking at publications indexed in Scopus, we detected research communities among Italian sociologists. We found that Italian sociologists are fractured in many disconnected groups. The giant connected component could be split into five main groups with a mix of three main disciplinary topics: sociology of culture and communication (present in two groups), economic sociology (present in three groups) and general sociology (present in three groups). By applying an exponential random graph model, we found that collaboration ties are mainly driven by theresearch interestsof these groups. Other factors, such aspreferential attachment,genderandaffiliation homophilyare also important, but the effect of gender fades away once other factors are controlled for. Our research shows the advantages of multi-level and temporal network analysis in revealing the complexity of scientific collaboration patterns. Show less
Community detection is often used to understand the structure of large and complex networks. One of the most popular algorithms for uncovering community structure is the so-called Louvain algorithm... Show moreCommunity detection is often used to understand the structure of large and complex networks. One of the most popular algorithms for uncovering community structure is the so-called Louvain algorithm. We show that this algorithm has a major defect that largely went unnoticed until now: the Louvain algorithm may yield arbitrarily badly connected communities. In the worst case, communities may even be disconnected, especially when running the algorithm iteratively. In our experimental analysis, we observe that up to 25% of the communities are badly connected and up to 16% are disconnected. To address this problem, we introduce the Leiden algorithm. We prove that the Leiden algorithm yields communities that are guaranteed to be connected. In addition, we prove that, when the Leiden algorithm is applied iteratively, it converges to a partition in which all subsets of all communities are locally optimally assigned. Furthermore, by relying on a fast local move approach, the Leiden algorithm runs faster than the Louvain algorithm. We demonstrate the performance of the Leiden algorithm for several benchmark and real-world networks. We find that the Leiden algorithm is faster than the Louvain algorithm and uncovers better partitions, in addition to providing explicit guarantees. Show less
When performing a national research assessment, some countries rely on citation metrics whereas others, such as the UK, primarily use peer review. In the influential Metric Tide report, a low... Show moreWhen performing a national research assessment, some countries rely on citation metrics whereas others, such as the UK, primarily use peer review. In the influential Metric Tide report, a low agreement between metrics and peer review in the UK Research Excellence Framework (REF) was found. However, earlier studies observed much higher agreement between metrics and peer review in the REF and argued in favour of using metrics. This shows that there is considerable ambiguity in the discussion on agreement between metrics and peer review. We provide clarity in this discussion by considering four important points: (1) the level of aggregation of the analysis; (2) the use of either a size-dependent or a size-independent perspective; (3) the suitability of different measures of agreement; and (4) the uncertainty in peer review. In the context of the REF, we argue that agreement between metrics and peer review should be assessed at the institutional level rather than at the publication level. Both a size-dependent and a size-independent perspective are relevant in the REF. The interpretation of correlations may be problematic and as an alternative we therefore use measures of agreement that are based on the absolute or relative differences between metrics and peer review. To get an idea of the uncertainty in peer review, we rely on a model to bootstrap peer review outcomes. We conclude that particularly in Physics, Clinical Medicine, and Public Health, metrics agree relatively well with peer review and may offer an alternative to peer review. Show less
Traag, V.A.; Malgarini, M.; Cicero, T.; Sarlo, S.; Waltman, L. 2018
A common element of all performance based research funding systems is the need to evaluate research. A recurrent question in this context is whether peer review and metrics tend to yield similar... Show moreA common element of all performance based research funding systems is the need to evaluate research. A recurrent question in this context is whether peer review and metrics tend to yield similar outcomes, or whether they differ substantially. We here study peer review uncertainty at the institutional level. We rely on data collected by ANVUR, the agency tasked with implementing the Italian research assessment exercise called VQR. We find that peer review agreement is generally higher at the institutional level than at the publication level. Similarly, correlations between peer review and metrics also tend to be higher at the institutional level. Finally, we find that the correlations between especially journal metrics and peer review are on par with correlations among two peer reviewers. Our results support the possibility of using metrics in combination with peer review for evaluation purposes. Show less
