Estimating the Proportion of False Null Hypotheses among a Large Number of Independently Tested Hypotheses

October, 2004
Report Number: 
Nicolai Meinshausen & John Rice

We consider the problem of estimating the number of false null hypotheses among a very large number of independently tested hypotheses, focusing on the situation in which the proportion of false null hypotheses is very small. We propose a family of methods for establishing lower $100(1-\alpha)\%$ confidence bounds for this proportion, based on the empirical distribution of the p-values of the tests. Methods in this family are then compared in terms of ability to consistently estimate the proportion by letting $\alpha \rightarrow 0$ as the number of hypothesis tests increases and the proportion decreases. This work is motivated by a signal detection problem occurring in astronomy.

PDF File: 
Postscript File: