In order to assess Microsoft Academic as a useful data source for evaluative bibliometrics it is crucial to know, if citation counts from Microsoft Academic could be used in common normalization... Show moreIn order to assess Microsoft Academic as a useful data source for evaluative bibliometrics it is crucial to know, if citation counts from Microsoft Academic could be used in common normalization procedures and whether the normalized scores agree with the scores calculated on the basis of established databases. To this end, we calculate the field-normalized citation scores of the publications of a computer science institute based on Microsoft Academic and the Web of Science and estimate the statistical concordance of the scores. Our results suggest that field-normalized citation scores can be calculated with Microsoft Academic and that these scores are in good agreement with the corresponding scores from the Web of Science. Show less
This short opinion paper seeks to cut through some the hype and outline some basic facts of what we know and do not know at the moment about “reproducibility” and “replication”. I will discuss... Show moreThis short opinion paper seeks to cut through some the hype and outline some basic facts of what we know and do not know at the moment about “reproducibility” and “replication”. I will discuss these issues and their implications for the field of Scientometrics, and I will briefly list some of the arguments brought forward for why data and code sharing should be universally promoted. However, my focus will be on the kernel of the current debate, the reliance of null hypothesis significance tests (NHST) for knowledge claims and the implications this have on reproducibility issues. The latter is as important for Scientometric and research evaluation studies, as it is for all other studies using such tests. Show less
In the social sciences and humanities book publishing takes a prominent role, both in terms of communicating with international peers and with a broader intelligentsia (Hicks, 2004; Verleysen &... Show moreIn the social sciences and humanities book publishing takes a prominent role, both in terms of communicating with international peers and with a broader intelligentsia (Hicks, 2004; Verleysen & Engels, 2014). Empirical evidence regarding the evolution of the share of scholarly book publications in the total volume of scholarly publications in a given country is rare. In this study we intend to fill this gap with an analysis of the comprehensive coverage data on the share of peer reviewed book publications (book chapters, edited volumes and monographs) that are available from Flanders and Slovenia for the period 2004 to 2015. We supplement these data with data on peer reviewed book chapters and monographs from Norway for the period 2005-2015 as well as data on all types of peer reviewed book publishing for the period 2009 to 2014 for Poland and 2011 to 2015 for Finland. Show less
In socially distributed models of knowledge production (Mode 2), internationalisation plays a major role in the production of scientific knowledge (Gibbons et al., 1994). Although there exist... Show moreIn socially distributed models of knowledge production (Mode 2), internationalisation plays a major role in the production of scientific knowledge (Gibbons et al., 1994). Although there exist profound disputes over the historical and conceptual accurateness of this model (Etzkowitz and Leydesdroff, 2000; Pestre, 2003), the increasing intensity and heterogeneity of international collaboration, a key component of internationalisation strategies, is a major point of agreement among scholars. Disagreement, however, still revolves around the methods that can be used to study internationalisation dynamics; whether they should be based on qualitative or quantitative techniques. The recent debate on the responsible use of scientometrics in research evaluation, I argue, could help shed some light into the use of scientometrics in the social study of science, technology and innovation (SSTI). More specifically, by arguing in favour of developing a multi-level analysis and a mixed method approach, I show how the study of international scientific collaboration offers an ideal setting where both qualitative and quantitative traditions can start a dialogue. This requires, nevertheless, understanding first the ontological and epistemological basis of scientometric indicators and the implications of its use in the social study of science. Show less
We analyze the role of leadership and scientific collaborative relationships in constituting the disciplinary specialization between countries and its research performance. Authorship order... Show moreWe analyze the role of leadership and scientific collaborative relationships in constituting the disciplinary specialization between countries and its research performance. Authorship order provides critical information for the allocation of reward, while collaboration enables researchers to expand the network of co-authors, institutions, and countries involved in the research. Along with these factors, a country’s profile orientation within the global scientific market become of great importance to the development of countries. As bibliographic data embedded such important information about the changes in the position of authors in the byline of publications and the disciplines involved in the research, we analyze these changes over time—using a Web of Science dataset—to explore the extent to which collaboration relationships impact leadership and specialization on the scientific workforce. Using this data, we discern the importance of domestic and international outputs in determining the disciplinary structure in scientific relationships in terms of publications and citations. We found that different types of leadership translate in different results in terms of relative specialization and citations. Overall results show that non-leading internationally collaborative papers reach higher values than leading international and domestic papers especially remarkable in terms of citations. Although in general, all regions increase their performance when collaborating with leading partners, the largest differences in research performance by leadership are located in countries with the lowest investment in R&D. Countries with the highest research investment are more likely to serve as leaders and garner higher specialization and citations when they lead (domestic and lead authorship). Comparative analyses of the role of specialization between countries can be useful for informing policies and motivating further collaboration relationships in the definitions of research agendas. Show less
In this paper we investigate how a publication indicator developed for the performance-based research funding system (PRFS) in Finland takes national language publications into account. Our... Show moreIn this paper we investigate how a publication indicator developed for the performance-based research funding system (PRFS) in Finland takes national language publications into account. Our analysis is based on 47423 peer-reviewed SSH outputs from 14 Finnish universities published in 2011-2016. SSH research community in Finland is increasingly concerned about the national language publishing. Incentives for English language journal publishing are attributed to the PRFS for allocating block grant annually to universities. In the Norwegian model adopted also in Finland, the weight of outputs in the funding-scheme is dependent on the quality index of publication channels. Our analysis shows that the rating of publication channels results in a fairly balanced representation of Finnish, Swedish and English language journal articles in the PRFS. The system is more favorable to English than Finnish and Swedish language book publications. Publications in other languages, however, are under-represented in the PRFS. The number of journal articles in Finnish has remained relatively stable. The number of Finnish book publications, however, is declining in both low and high rated outlets. We speculate that in addition to the PRFS, publishing patterns are influenced by increased international competition for positions and project funding, as well as other factors than research evaluation. Show less
Zagovora, O.; Weller, K.; Janosov, M.; Wagner, C.; Peters, I. 2018
Do only major scientific breakthroughs hit the news and social media, or does a `catchy' title help to attract public attention? How strong is the connection between the importance of a scientific... Show moreDo only major scientific breakthroughs hit the news and social media, or does a `catchy' title help to attract public attention? How strong is the connection between the importance of a scientific paper and the (social) media attention it receives? In this study we investigate these questions by analysing the relationship between the observed attention and certain characteristics of scientific papers from two major multidisciplinary journals: Nature Communication (NC) and Proceedings of the National Academy of Sciences (PNAS). We describe papers by features based on the linguistic properties of their titles and centrality measures of their authors in their co-authorship network. We identify linguistic features and collaboration patterns that might be indicators for future attention, and are characteristic to different journals, research disciplines, and media sources. Show less
This paper examines the feasibility of replacing the resource-intensive and contested expert review panels in the British national research evaluation – called REF (Research Excellence Framework) –... Show moreThis paper examines the feasibility of replacing the resource-intensive and contested expert review panels in the British national research evaluation – called REF (Research Excellence Framework) – with a much simpler exercise based on metrics. Using the free Publish or Perish software to source citation data from Microsoft Academic (MA), I was able to create a citation-based ranking of British universities that correlates with the REF power ranking at 0.97. I thus suggest that metrics should be considered seriously as an alternative to peer review. It doesn’t mean that we should give up completely on evaluating research quality through peer review. But letting metrics do the bulk of the work frees up a staggering amount of time and resources that would allow us to come up with more creative and meaningful ways to build in a credible quality component in the British national research assessment. Show less
Araújo Almeida, B. de; Xavier, P.; Barreto, M. 2018
The deployment of administrative databases for research purposes holds great potential. Administrative data becomes even more useful when linked to other datasets, making it possible to elucidate... Show moreThe deployment of administrative databases for research purposes holds great potential. Administrative data becomes even more useful when linked to other datasets, making it possible to elucidate the effects of combined factors that could potentially impact the health of populations. On the other hand, the access, use and reuse of these administrative datasets, principally those containing personally identifiable information, are all topics being widely discussed nowadays in an effort to maintain confidentiality, protect individuals’ privacy, respect terms of consent and manage security or other risks. For these reasons, we believe that our data management experience at the Fiocruz Center of Data and Knowledge Integration for Health (CIDACS) in Brazil can make a significant contribution by providing high-quality linked data for research purposes and generating evidence for public health policy. Show less
Maps depicting the structure of science help us understand the development of science and technology. However, as it is difficult to apply inter-citation and co-citation analysis to recently... Show moreMaps depicting the structure of science help us understand the development of science and technology. However, as it is difficult to apply inter-citation and co-citation analysis to recently published papers and ongoing projects that have few or no references, our previous work developed a content-based map by locating research papers and funding projects using word/document embedding. Because difficulties arise when comparing the content-based map in different languages, this paper improves our content-based map by developing a method for generating multi-dimensional vectors in the same space from cross-lingual (English and Japanese) documents. Using 1,000 IEEE papers, we confirmed a similarity of 0.76 for matching bilingual contents. Finally, we constructed a map from 34,000 projects of the National Science Foundation and Japan Society for the Promotion of Science from 2012 to 2015, and we indicate the findings obtained from a comparison of the US-Japan funded projects. Show less
Expansion of government R & D budgets on promoting electric vehicle (EV) adoption and charging infrastructure development is likely to continue to be a key component of ecological innovation... Show moreExpansion of government R & D budgets on promoting electric vehicle (EV) adoption and charging infrastructure development is likely to continue to be a key component of ecological innovation policies. Using an original data set of non-patent literature (NPL) references extracted from patent documents pertaining EV charging technologies, this paper provides new evidence on the flows of knowledge with or without a scientific contribution from the business sector. Three main questions are addressed in this paper for measuring the value of knowledge produced by firms, which not only contributes towards a better understanding of EV but serves the purpose of fostering more partnerships and unlocking further investments in research. First, what information is most useful to the technological development? Even firms are increasingly encouraged to engage in EV innovation process, a relatively profound influence on knowledge transfer has not be exercised, especially in generating applied technologies measured by redefined average NPL citation compared to academic institutions. Patents with firm NPL have a special focus on inorganic chemistry and nanotechnology except as common issues identified related to climate change mitigation and energy storage. Second, which kind of firm’s contribution produces the most valuable research? The university-firm research collaborations have captured more attention from science to technology while knowledge produced solely by firms has been transferred to a broader distribution in geography. Finally, how scientific knowledge is commercialised? Patents with firm NPLs, in particular the one regarding networked infrastructure and energy generating have been transferred more frequently to companies and universities residing in the US, Japan, Canada and Germany between 2010 and 2014. However, patented technologies of electrical distribution network and charging batteries with non-firm NPLs are mainly assigned to companies in France and Korea between 2008 and 2013. The role of firm in knowledge and technology transfer needs to be further explored in a border technological field notwithstanding the gaps in NPL citation compared to academic institutions. Show less
In recent years, increasing attention is being drawn to the analysis of scientific career path using the indicators of academic career in career studies. Bibliometric methods play an increasing... Show moreIn recent years, increasing attention is being drawn to the analysis of scientific career path using the indicators of academic career in career studies. Bibliometric methods play an increasing role in scientific career path analysis and its evaluation but an integrative approach is still missing to characterise the multiple aspects of scientific career. The goal of this paper is (1) to demonstrate an attempt to synthetize the prexisting bibliometric indicators, setting up a system of measurements providing a rich scientometric profile on academic careers, and (2) to present a case study on a Hungarian sample of researchers, including an empirical career typology based on our indicator system. Show less
The public sector is an important source of agricultural research as many of the agricultural producers are individual farmers who cannot bear the cost of research and development. Public... Show moreThe public sector is an important source of agricultural research as many of the agricultural producers are individual farmers who cannot bear the cost of research and development. Public institutes for experimentation and research called Kosetsushi help producers in agriculture and manufacturing improve productivity through technology transfer activities, thereby constituting an important component of regional innovation systems in Japan. This study establishes panel data of agricultural research input and output (new variety of a plant) by Kosetsushi and examines whether their research activities are responsive to local needs and which type of research effort is conducive to the promotion of agricultural product innovation measured as registration of new varieties. Estimation results reveal significant variations across plants in the impacts of agricultural clusters on research on that plant. The impact of research on plant breeding on product innovation also varies across plants. Policy implications of the major findings are discussed. Show less
Haunschild, R.; Marx, M.; French, B.; Bornmann, L. 2018
Three different approaches of field-categorization are currently (mainly) used for normalizing citation impact without a clear preference for one alternative: (1) journal sets, (2) intellectual... Show moreThree different approaches of field-categorization are currently (mainly) used for normalizing citation impact without a clear preference for one alternative: (1) journal sets, (2) intellectual assignments, and (3) citation relations. In this study, we compare normalized citation scores, which have been calculated based on the three approaches to build reference sets. We are interested whether they lead to the same, similar, or different scores for the same papers – if the formula of calculating the scores is held constant. This study focusses on chemistry and related sciences, because we have access to a comprehensive dataset from Chemical Abstracts Service (CAS). The results show that normalized scores based on intellectual field assignments are more in agreement with scores based on journal sets than with scores based on citation relations. Thus, one can expect more similar scores based on intellectual assignments and journal sets than on citation relations. Show less
Responsible use of scientometrics in research evaluations is heavily debated. My aim in this opinion paper is to draw attention to the importance of distinguishing between different evaluative... Show moreResponsible use of scientometrics in research evaluations is heavily debated. My aim in this opinion paper is to draw attention to the importance of distinguishing between different evaluative settings. Principles for responsible use of scientometrics are dependent on the nature of the evaluative setting. What is responsible in one setting may not be responsible at all in a different setting. Recognizing that there is no one size that fits all leads to insights that may help to enhance responsible use of scientometrics. To illustrate the importance of distinguishing between different evaluative settings, my focus is on the distinction between micro-level and macro-level research evaluation. Show less
We explore whether and how mapping bibliometric methods combined with other data sources (mentions in news and policy) can also be useful for mapping potential societal engagement of research... Show moreWe explore whether and how mapping bibliometric methods combined with other data sources (mentions in news and policy) can also be useful for mapping potential societal engagement of research fields. We use as a case study agricultural research, broadly defined, since this is a field that has long being recognised as problematic for conventional bibliometric assessment, particularly in developing contexts. On the one hand, we use conventional bibliometric data such as percentage of internal coverage, non-English publications, co-word mapping and the fine-grained classification of science provided by article-level clustering. On the other hand, we explore relative frequency of mentions in policy documents and news items.The comparison of patterns across research areas suggests those areas that may be more directly engaged with societal actors. Show less
Cumulative advantage – commonly known as the Matthew Effect – influences scientific output and careers. Given the challenge and uncertainty of gauging the quality of new scientific research,... Show moreCumulative advantage – commonly known as the Matthew Effect – influences scientific output and careers. Given the challenge and uncertainty of gauging the quality of new scientific research, evaluators and gatekeepers often possess incentives to prefer the work of established scientists. Such preferences breach scientific norms of fairness, and can yield suboptimal research outcomes. This article analyzes repeat authors as an exemplar of the Matthew Effect. Although a scientist publishing in the same journal multiple times is rare within individual careers, the phenomenon is relatively common at the level of scientific journals. Using publication data for 347 economics journals from 1980-2016, we analyze whether articles written by repeat authors tend to fare better or worse than less-experienced authors. Random effects models show a curvilinear (inverted U-shaped) relationship between repeat authorship and citation impact. In these models, citation impact peaks at the fourth repeat publication, suggesting both liabilities of newness and liabilities of senescence in science. Fixed effects models show that within individual scientific careers, authors tend to be most impactful with their debut publication, then experience declining impact with each subsequent repeat authorship. Implications for innovation incentives for scientists and gatekeepers alike are discussed. Show less
