This is a small-scale explorative study in an effort to assess reproducibility issues specific to scientometrics research. This effort is motivated by the desire to generate empirical data to... Show moreThis is a small-scale explorative study in an effort to assess reproducibility issues specific to scientometrics research. This effort is motivated by the desire to generate empirical data to inform debates about reproducibility in scientometrics. Rather than attempt to reproduce studies, we explore how we might assess ‘in principle’ reproducibility based on a critical review of the content of published papers. While the first part of the study focuses on direct reproducibility - that is the ability to reproduce the specific evidence produced by an original study using the same data, methods, and procedures, the second part is dedicated to conceptual reproducibility - that is the robustness of knowledge claims towards verification by an alternative approach using different data, methods and procedures. The study is exploratory: it investigates only a very limited number of publications and serves us to develop instruments for identifying potential reproducibility issues of published studies: These are a categorization of study types and a taxonomy of threats to reproducibility. We work with a select sample of five publications in scientometrics covering a variation of study types of theoretical, methodological, and empirical nature. Based on observations made during our exploratory review, we conclude with open questions to consider when adressing threats to reproducibility in scientometric research that are intended for discussion at the special track on ‘Reproducibility in Scientometrics’ at STI2018 in Leiden. Show less
The unprecedented growth of Chinese publications, understood as a macro level intervention on the global bibliometric universe, is analysed on its causal effects on the bibliometric measurement... Show moreThe unprecedented growth of Chinese publications, understood as a macro level intervention on the global bibliometric universe, is analysed on its causal effects on the bibliometric measurement process. By their size and particular bibliometric characteristics these publications might not be considered as marginal, but impose a structural change upon all countries contributing to the global science system. A causal analysis is conducted based upon the statistical approach of treatment effects models. In detail a counterfactual bibliometric world without Chinese publications is artificially constructed and contrasted with the actual equivalent. We observe that the Chinese publications cause an increase in the expected citation counts as they included above average numbers of references. At the same time the citations of Chinese publications are especially focused upon scientifically leading, as well as Asian countries, resulting in an increase in their measured impact. But as any identification of a causal effect via statistical means arises as an interpretation, we subsequently discuss the limitation of our approach namely the imperfect knowledge on the counterfactual state and any implications arising for further causal evaluations, i.e. the Chinese publication growth illustrates only one of several structural effects concurrently driving the evolution of the bibliometric universe. Show less
