We consider the class of all multiple testing methods controlling tail probabilities of the false discovery proportion, either for one random set or simultaneously for many such sets. This class... Show moreWe consider the class of all multiple testing methods controlling tail probabilities of the false discovery proportion, either for one random set or simultaneously for many such sets. This class encompasses methods controlling familywise error rate, generalized familywise error rate, false discovery exceedance, joint error rate, simultaneous control of all false discovery proportions, and others, as well as gene set testing in genomics and cluster inference in neuroimaging. We show that all such methods are either equivalent to a closed testing procedure, or are uniformly improved by one. Moreover, we show that a closed testing method is admissible if and only if all its local tests are admissible. This implies that, when designing methods, it is sufficient to restrict attention to closed testing. We demonstrate the practical usefulness of this design principle by obtaining more informative inferences from the method of higher criticism, and by constructing a uniform improvement of a recently proposed method. Show less
Modern day technology allows us to collect and store more data than ever before. Although the abundance of data offers great opportunities with respect to formulating and testing all kinds of... Show moreModern day technology allows us to collect and store more data than ever before. Although the abundance of data offers great opportunities with respect to formulating and testing all kinds of hypotheses and with respect to discovering and modeling patterns that can be used for future prediction, these opportunities do not come without risks. If the data is explored in a too naive manner, there is a very large probability that, at least part of, the actual findings will not be reproducible in future research. When the number of hypotheses or the number of parameters within a model increases, the possibility of actually ending up in such a situation increases as well. Profiting from all available data, while preventing chance findings from occurring, will call for advanced statistical techniques. The development of such techniques is the subject of this thesis. We focus both on multiple hypothesis testing and on the construction of prediction models. Throughout the thesis, emphasis is placed on the development of methods that are efficient, both from a power perspective and from the perspective of computational feasibility. Show less
