Replication of published results is crucial for ensuring the robustness and self-correction of research, yet replications are scarce in many fields. Replicating researchers will therefore often... Show moreReplication of published results is crucial for ensuring the robustness and self-correction of research, yet replications are scarce in many fields. Replicating researchers will therefore often have to decide which of several relevant candidates to target for replication. Formal strategies for efficient study selection have been proposed, but none have been explored for practical feasibility - a prerequisite for validation. Here we move one step closer to efficient replication study selection by exploring the feasibility of a particular selection strategy that estimates replication value as a function of citation impact and sample size (Isager, van 't Veer, & Lakens, 2021). We tested our strategy on a sample of fMRI studies in social neuroscience. We first report our efforts to generate a representative candidate set of replication targets. We then explore the feasibility and reliability of estimating replication value for the targets in our set, resulting in a dataset of 1358 studies ranked on their value of prioritising them for replication. In addition, we carefully examine possible measures, test auxiliary assumptions, and identify boundary conditions of measuring value and uncertainty. We end our report by discussing how future validation studies might be designed. Our study demonstrates the importance of investigating how to implement study selection strategies in practice. Our sample and study design can be extended to explore the feasibility of other formal study selection strategies that have been proposed. Show less
Pittelkow, M.-M.; Field, S.M.; Isager, P.M.; Veer, A.E. van 't; Anderson, T.; Cole, S.N.; ... ; Ravenzwaaij, D. van 2023
Robust scientific knowledge is contingent upon replication of original findings. However, replicating researchers are constrained by resources, and will almost always have to choose one replication... Show moreRobust scientific knowledge is contingent upon replication of original findings. However, replicating researchers are constrained by resources, and will almost always have to choose one replication effort to focus on from a set of potential candidates. To select a candidate efficiently in these cases, we need methods for deciding which out of all candidates considered would be the most useful to replicate, given some overall goal researchers wish to achieve. In this article we assume that the overall goal researchers wish to achieve is to maximize the utility gained by conducting the replication study. We then propose a general rule for study selection in replication research based on the replication value of the set of claims considered for replication. The replication value of a claim is defined as the maximum expected utility we could gain by conducting a replication of the claim, and is a function of (a) the value of being certain about the claim, and (b) uncertainty about the claim based on current evidence. We formalize this definition in terms of a causal decision model, utilizing concepts from decision theory and causal graph modeling. We discuss the validity of using replication value as a measure of expected utility gain, and we suggest approaches for deriving quantitative estimates of replication value. Our goal in this article is not to define concrete guidelines for study selection, but to provide the necessary theoretical foundations on which such concrete guidelines could be built.Translational Abstract Replication-redoing a study using the same procedures-is an important part of checking the robustness of claims in the psychological literature. The practice of replicating original studies has been woefully devalued for many years, but this is now changing. Recent calls for improving the quality of research in psychology has generated a surge of interest in funding, conducting, and publishing replication studies. Because many studies have never been replicated, and researchers have limited time and money to perform replication studies, researchers must decide which studies are the most important to replicate. This way scientists learn the most, given limited resources. In this article, we lay out what it means to think about what is the most important thing to replicate, and we propose a general decision rule for picking a study to replicate. That rule depends on a concept we call replication value. Replication value is a function of the importance of the study, and how uncertain we are about the findings. In this article we explain how researchers can think precisely about the value of replication studies. We then discuss when and how it makes sense to use replication value as a measure of how valuable a replication study would be, and we discuss factors that funders, journals, or scientists could consider when determining how valuable a replication study is. Show less
Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance... Show moreData analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses. Show less
Botvinik-Nezer, R.; Holzmeister, F.; Camerer, C.F.; Dreber, A.; Huber, J.; Johannesson, M.; ... ; Schonberg, T. 2020
Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance... Show moreData analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses. Show less
Botvinik-Nezer, R.; Holzmeister, F.; Camerer, C.F.; Dreber, A.; Huber, J.; Johannesson, M.; ... ; Schonberg, T. 2020
Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance... Show moreData analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses. Show less
Isager, P.M.; Van Aert, R.C.M.; Bahník, S.; Brandt, M.J.; DeSoto, K.A.; Giner-Sorolla, R. Krueger, J.; ... ; Lakens, D. 2020
Preregistration clarifies the distinction between planned and unplanned research by reducing unnoticed flexibility. This improves credibility of findings and calibration of uncertainty. However,... Show morePreregistration clarifies the distinction between planned and unplanned research by reducing unnoticed flexibility. This improves credibility of findings and calibration of uncertainty. However, making decisions before conducting analyses requires practice. During report writing, respecting both what was planned and what actually happened requires good judgment and humility in making claims. Show less
The underlying mechanisms of paternal responses to infant signals are poorly understood. Vasopressin has previously been proposed to affect these responses. Using a double-blind, placebo-controlled... Show moreThe underlying mechanisms of paternal responses to infant signals are poorly understood. Vasopressin has previously been proposed to affect these responses. Using a double-blind, placebo-controlled, within-subject design (N = 25 expectant fathers), we examined the effect of vasopressin administration on the use of excessive handgrip force during exposure to infant crying versus matched control sounds, while participants saw morphed images representing their own infant versus an unknown infant. We found that, compared to placebo, AVP administration elicited more excessive force while viewing an unknown infant image compared to viewing the image representing one’s own infant, while the reverse was true under placebo. The results are discussed in light of vasopressin’s role in parenting and parental protection among human fathers. Show less
In this lab-based experiment (N = 185, Tilburg University students) we tested the effect of anticipatory stress on moral condemnation. The data covers severity ratings for vignettes of two content... Show moreIn this lab-based experiment (N = 185, Tilburg University students) we tested the effect of anticipatory stress on moral condemnation. The data covers severity ratings for vignettes of two content types: vignettes with an inherent disgust-eliciting element (e.g., eating human flesh) and without (e.g., lying on a resume), filled out on computers using the survey platform Qualtrics. Participants in the anticipatory stress condition rated the vignettes as more morally wrong, and disgust-eliciting vignettes were rated as more morally wrong. No moderation by disgust content was found. Private Body Consciousness (PBC) was positively associated with condemnation for disgust-eliciting vignettes (but not with non-disgust-eliciting vignettes). The data can be used, for example, in research on incidental vs. inherent emotions, to identify the strength of induced emotions on judgments, and to identify moderators (e.g., PBC). Show less
Perceiving potential threat to an infant and responding to it is crucial for offspring survival and parent–child bonding. Using a combination of functional magnetic resonance imaging and multi... Show morePerceiving potential threat to an infant and responding to it is crucial for offspring survival and parent–child bonding. Using a combination of functional magnetic resonance imaging and multi-informant reports, this longitudinal study explores the neural basis for paternal responses to threat to infants pre-natally (N = 21) and early post-natally (n = 17). Participants viewed videos showing an infant in danger and matched control videos, while instructed to imagine that the infant was their own or someone else’s. Effects were found for infant-threatening vs neutral situations in the amygdala (region-of-interest analyses) and in clusters spanning cortical and subcortical areas (whole-brain analyses). An interaction effect revealed increased activation for own (vs unknown) infants in threatening (vs neutral) situations in bilateral motor areas, possibly indicating preparation for action. Post-natal activation patterns were similar; however, in part of the superior frontal gyrus the distinction between threat to own and unknown infant faded. Fathers showing more protective behavior in daily life recruited part of the frontal pole more when confronted with threat to their own vs an unknown infant. This exploratory study is the first to describe neural mechanisms involved in paternal protection and provides a basis for future work on fathers’ protective parenting. Show less
Thijssen, S.; Veer, A.E. van 't; Witteman, J.; Meijer, W.M.; Van IJzendoorn, M.H.; Bakermans-Kranenburg, M.J. 2018
