Bibliometric uncertainty: data quality, reliability, and validity

Abstract

Bibliometric methods and bibliometric indicators are important, albeit disputed, tools in empirical studies of science systems and in research evaluation. Their validity and acceptance is partly due to their statistical scope, allowing for analyses at large scales, such as entire countries, for which the cost of employing qualitative methods is too high. Bibliometric methods are deeply data-driven which makes the trustworthiness of the statements and interpretations derived from bibliometric studies very data-dependent. Science studies researchers have long recognized that poor quality data is a possible threat to informative scientific outcomes and solid, defensible quantitative assessment of research. They have consequently put considerable effort into empirically investigating the quality of their data although this has not yet precipitated the emergence of a consolidated systematic research program in its own right. Scientometricians and bibliometricians have studied such data quality topics and author name disambiguation, organization name disambiguation, completeness and accuracy of various bibliographic fields such as cited references, identifiers (DOIs), and assigned document types and categories of subject classification systems. Not infrequently, the practical consequences of the observed error rates have remained unclear. How to express the amount of uncertainty in bibliometric that are caused by inaccuracies in the basic data has also remained elusive. The research of this dissertation extends the knowledge on error rates of specific data attributes, namely document types and citation rates, and studies also the performance of different organization disambiguation systems. It further examines the standard methodology of citation analysis of using nominal year-based citation windows and find this to be biased. In all studies, emphasis is put on the real-world implications of these deficiencies and possible improvements. The final part introduces a novel framework for the propagation and reporting of uncertainties that arise from such data quality issues. I show how inaccuracies in input data can be used to derive realistic error margins for bibliometric indicator result values that quantify their uncertainty due to statistical error in the data. The main result is that bibliometric indicator values are commonly affected by the error sources that were studied that small to moderate differences in indicator values –even for aggregate units such as research groups, departments, universities– might be attributed to data inaccuracies rather than actual performance differences. The implications and consequences of these findings are discussed and recommendations for improvements of current practice are derived.