Leiden University Scholarly Publications

Persistent URL of this record https://hdl.handle.net/1887/3277594

In Collections

This item can be found in the following collections:

Zwet, E.W. van; Cator, E.A. (2021)

The significance filter, the winner's curse and the need to shrink

Article / Letter to editor
The "significance filter" refers to focusing exclusively on statistically significant results. Since frequentist properties such as unbiasedness and coverage are valid only before the data have been observed, there are no guarantees if we condition on significance. In fact, the significance filter leads to overestimation of the magnitude of the parameter, which has been called the "winner's curse." It can also lead to undercoverage of the confidence interval. Moreover, these problems become more severe if the power is low. These issues clearly deserve our attention. They have been studied mostly through empirical observation and simulation, while there are relatively few mathematical results. Here we study them both from the frequentist and the Bayesian perspective. We prove that the relative bias of the magnitude is a decreasing function of the power and that the usual confidence interval undercovers when the power is less than 50%. We conclude that it is important to apply the...Show moreThe "significance filter" refers to focusing exclusively on statistically significant results. Since frequentist properties such as unbiasedness and coverage are valid only before the data have been observed, there are no guarantees if we condition on significance. In fact, the significance filter leads to overestimation of the magnitude of the parameter, which has been called the "winner's curse." It can also lead to undercoverage of the confidence interval. Moreover, these problems become more severe if the power is low. These issues clearly deserve our attention. They have been studied mostly through empirical observation and simulation, while there are relatively few mathematical results. Here we study them both from the frequentist and the Bayesian perspective. We prove that the relative bias of the magnitude is a decreasing function of the power and that the usual confidence interval undercovers when the power is less than 50%. We conclude that it is important to apply the appropriate amount of shrinkage to counter the winner's curse.Show less
bias
exaggeration ratio
shrinkage
type M error
All authors
Zwet, E.W. van; Cator, E.A.
Date
2021-04-04
Journal
Statistica Neerlandica
Volume
75
Issue
4
Pages
437 - 452
DOI
doi:10.1111/stan.12241
Link
https://onlinelibrary.wiley.com/doi/10.1111/stan.12241