Data-dependent analytical choices relying on NHST should not be trusted!

Abstract

This short opinion paper seeks to cut through some the hype and outline some basic facts of what we know and do not know at the moment about “reproducibility” and “replication”. I will discuss these issues and their implications for the field of Scientometrics, and I will briefly list some of the arguments brought forward for why data and code sharing should be universally promoted. However, my focus will be on the kernel of the current debate, the reliance of null hypothesis significance tests (NHST) for knowledge claims and the implications this have on reproducibility issues. The latter is as important for Scientometric and research evaluation studies, as it is for all other studies using such tests.