A systematic comparison of methods for combining p-values from independent tests

Six methods are studied for combining p-values from independent tests into a new test of the combined hypothesis. The methods-minimum (The Method of Statistics, Williams and Norgate, London, 1931), chi-square (2)(Statistical Methods for Research Workers, 4th Edition, Oliver and Boyd, London, 1932), normal (Magyar Tudomanyos Akademia. Matematikai Kutato Intezetenek Kozlemenyei 3 (1958) 1971), maximum (Wilkinson, Psycholog. Bull. 48 (1951) 156), uniform (J. Pyschol. 80 (1972) 351), and logistic (in: Rustagi (Ed.), Symposium on Optimizing Methods in Statistics, Academic Press, New York, 1979, pp. 345-366)-are compared heuristically and through simulation. Plots of the rejection regions for combining two tests reveal much about the tests' relative strengths. The simulations compare methods using different numbers of tests, different patterns of evidence against the combined null hypothesis, and different total strengths of the evidence, allowing broader recommendations than have been made from past simulations. The results indicate that the most difficult kind of problem for a combined test is one in which the total evidence against the combined null is concentrated in one or very few of the tests being combined. For this case alone is the minimum combining function useful. The normal combining function does well in problems where evidence against the combined null is spread among more than a small fraction of the individual tests, or when the total evidence is weak. The chi-square (2) does best when the evidence is at least moderately strong and is concentrated in a relatively small fraction of the individual tests. The logistic combination provides a compromise between these two. The maximum and uniform combinations have generally very poor power and cannot be recommended for use. (C) 2003 Elsevier B.V. All rights reserved.