We address the question of why global university rankings should not be used for research evaluation. To answer this question, we analyze four groups of literature (academic vs non-academic... Show moreWe address the question of why global university rankings should not be used for research evaluation. To answer this question, we analyze four groups of literature (academic vs non-academic literature, English-language vs Russian-language literature). The analysis shows that most researchers agree that rankings should not be used to evaluate research. However, they are still used for these purposes directly or indirectly, although recent developments give us hope for a change in the situation in the near future. Show less
Several initiatives have been taken to promote the openly availability of bibliographic metadata of scholarly publications in Crossref. We present an up-to-date overview of the availability of six... Show moreSeveral initiatives have been taken to promote the openly availability of bibliographic metadata of scholarly publications in Crossref. We present an up-to-date overview of the availability of six metadata elements in Crossref: reference lists, abstracts, ORCIDs, author affiliations, funding information, and license information. Our analysis shows that the availability of these metadata elements has improved over time. However, it also shows that many publishers need to make additional efforts to realize full openness of bibliographic metadata. To illustrate the value of open metadata, we use the metadata in Crossref to construct and visualize a large citation network of scholarly journals. Show less
Modern natural language processing techniques have given rise to embedding techniques that can represent documents based on their content or context, and several papers have operationalized these... Show moreModern natural language processing techniques have given rise to embedding techniques that can represent documents based on their content or context, and several papers have operationalized these to perform bibliometric tasks. The relationship between these embeddings and conventional citation based or title and abstract based mappings remains unclear. Contrary to citation-based or term-based relatedness, embedding-based relatedness is not immediately interpretable. We consider four embedding-derived publication relatedness measures, based on: 1) word2vec embeddings of citation labels, sentence embeddings using 2) BERT and 3) SciBERT, and 4) title and abstract embeddings using SPECTER, and compare them with conventional bibliometric publication relatedness measures derived from citation relations and title and abstract noun phrases. We show that there is stronger overlap between these embedding-derived relatedness measures and citation-based relatedness than with title and abstract noun phrase-based relatedness, and that embedding-derived relatedness measures outperform conventional techniques when used to cluster publications cited with the same citation intent. Show less
The Arabic Citation Index (ARCI) was launched in 2020. This study gives an overview of the scientific literature available in this new database. I analyse ARCI by using metadata available in... Show moreThe Arabic Citation Index (ARCI) was launched in 2020. This study gives an overview of the scientific literature available in this new database. I analyse ARCI by using metadata available in scientific publications to characterize its coverage. First, I describe the data and the methods used in the analyses. As of October 2020, ARCI indexed 65,208 records covering the 2015-2019 period. Second, I explore the literature distributions at various levels (research domains, countries, languages, open access). Close to 99% of documents indexed are articles. Results reveal the concentration of publications in the Arts & Humanities and Social Sciences fields. Most journals indexed in ARCI are currently published from Egypt, Algeria, Iraq, Jordan and Saudi Arabia. Around 7% of publications in ARCI are published in languages other than Arabic. Then, I use an unsupervised machine learning model, LDA (Latent Dirichlet Allocation) and the text mining algorithms of VOSviewer to uncover the main topics in ARCI. These methods are particularly useful to better understand the topical structure of ARCI. Finally, I suggest few research opportunities after discussing the results of this study. Show less
Ferreira, M.R.; Bascur Cifuentes, J.P.; Costas Comesana, R. 2021
As bibliographic reference managers like Mendeley made their data openly available, it became possible to track where in the world research was being saved from. This data offered the opportunity... Show moreAs bibliographic reference managers like Mendeley made their data openly available, it became possible to track where in the world research was being saved from. This data offered the opportunity to better understand how research circulates at a global scale with measures that go beyond citations. This paper explores this circulation by studying fluctuations in rankings between countries when they are based on mean normalized citation scores (MNCS) or on mean normalized Mendeley readership scores (MNRS). Results show that both indicators are moderately correlated at the country level, but that countries from the Global South (namely African and South American countries) perform better when ranked by Mendeley readership than by citations. In addition, publications from South America and Africa tend to have a lower citation impact compared to those from Europe and North America, even when compared with publications that have the same number of readers. These results suggest that the indicator chosen (i.e., citations or Mendeley readers) creates different (dis)advantages among scholarly actors (e.g. countries, research organizations, journals, etc.). It also hints at the need to establish evaluation frameworks that consider that different metrics play different roles across institutional and geographical boundaries. We conclude by proposing further ways of exploring these metrics. Show less
In the context of a Dutch university medical center shifting from a largely bibliometrics-based form of performance evaluation, or ‘academic excellence’ valuation regime, towards evaluation based... Show moreIn the context of a Dutch university medical center shifting from a largely bibliometrics-based form of performance evaluation, or ‘academic excellence’ valuation regime, towards evaluation based on notions of ‘societal relevance’, this paper presents the findings of an ongoing ethnographic engagement with several biomedical research groups, designed to trace the ways in which variable regimes of valuation can take on a constitutive role in knowledge production processes. Whereas the Journal Impact Factor remains to be a key metric embedded in various research practices, researchers increasingly put effort into making ‘society’ present within their research work as to ensure its societal relevance and make it visible in a context of changing organization and funding infrastructures. The variable forms of this presence have a limited role in the constitution of knowledge, but the time, effort and resources needed to realize them can bear a complex relation to crucial choices and practices of research. Show less
Dagienė, E.; Kriščiūnas, A.; Tautkevičienė, G.; Maskeliūnas, S. 2019
For many years, scholarly books have been and continue to be an important channel of scholarly communication and a unit for research assessment. Books as a mean of scholarly publication are more... Show moreFor many years, scholarly books have been and continue to be an important channel of scholarly communication and a unit for research assessment. Books as a mean of scholarly publication are more frequently used in social sciences and humanities (SSH), yet remain essential for communication in science, technology and medicine (STM) (Bonaccorsi et al., 2017). However, in the context of research assessment, books and especially monographs are extensively discussed as an output in crisis due to the threats arising in relation to their quality and production (Basili and Lanzillo, 2018). In this work, we present a comprehensive and still ongoing study on the assessment of books in Lithuania to fill in the gaps in international knowledge on that topic. Show less
The Brazilian System of Graduate Courses counts with two different modalities of programs: academic and professional. The latter one was authorized by the Brazilian Ministry of Education in 1998,... Show moreThe Brazilian System of Graduate Courses counts with two different modalities of programs: academic and professional. The latter one was authorized by the Brazilian Ministry of Education in 1998, at first only for master’s courses. The idea to start this new modality solely on the first level was to allow the academic community to acquire the necessary experience with such courses, since they should focus on bringing scientific methodology to the productive sector, with problem-solving as one of its primary goals. Almost 20 years after they were first authorized, there are already 782 of these programs in Brazil, almost 20% of the total number of master’s courses currently active. Considering this development, in 2017 the Ministry of Education authorized the expansion of the new modality for the doctoral level as well, and 90 proposals for such courses have already been sent to CAPES, the agency in charge for the accreditation and evaluation of every graduate course in the country. Since the opening of the new professional doctorates is oncoming, the goal of this research is to investigate if the different approach expected from such courses has been materialized in the scientific production from professional master’s courses, when compared to that originated from the academic ones. Show less
Brandão, A.; Villela, A.; Oliveira, C.; Brasil, A.; Santos, A. 2018
Twitter users tweeting scholarly publications from different countries have been analysed. The aim is to explore how visible are different countries on Twitter (based on their self-assigned geo... Show moreTwitter users tweeting scholarly publications from different countries have been analysed. The aim is to explore how visible are different countries on Twitter (based on their self-assigned geo-locations obtained from altmetric.com) in comparison to their output size in the Web of Science. Some indicators such as Twitter presence and activity (such as number of user’s accounts, number of tweets, and number of publications tweeted) have been analysed for each country. Finally, the relationship between Twitter activity indicators and some demographic indicators (such as country’s population, education, internet users, ICT use and access) will be explored and potential factors affecting country’s activity on Twitter will be discussed. Exploring how technological access and development (technology orientation) of a country affects its scholarly twitter usage (scientific orientation) will help in interpreting country’s activity and in understanding the reasons why some countries are underrepresented on Twitter. Identification of such factors could help in gaining some insights on important barriers and limitations that may have an effect on usage of scholarly social media platforms by users from different countries and on considering ‘altmetric divide’ for any altmetrics applications at the country level. Show less
The country of authors of 5,9 million Web of Science (WoS) publications with DOI from the years 2012 to 2015 have been compared with the country of Twitter users tweeting these WoS publications in... Show moreThe country of authors of 5,9 million Web of Science (WoS) publications with DOI from the years 2012 to 2015 have been compared with the country of Twitter users tweeting these WoS publications in order to study the main scholarly users of Twitter across 10 different countries. For this purpose, the visibility of country’s publications in the WoS and geographical distribution of Twitter users tweeting WoS publications have been analysed. The aim is to study how do they differ and what are their preference in tweeting their own vs. other country’s publication. The findings show that in general, US and UK with the highest proportion of outputs in the WoS, are among the main users of Twitter as well. Moreover, except for US, users tweet publications affiliated to other country more than those from their own country. Also, similar to WoS, it seems that altmetric providers are not free of international biases in their coverage and collection of metrics. Finally, various possible reasons on why publications from some countries attract more Twitter users than others have been discussed. Show less
