Journal
BIOINFORMATICS
Volume 26, Issue 16, Pages 1990-1998Publisher
OXFORD UNIV PRESS
DOI: 10.1093/bioinformatics/btq323
Keywords
-
Categories
Funding
- German Science Foundation (DFG Einzelforderung) [BO3139/2-1]
- LMU
- French-Bavarian Cooperation Center for Universities (CCUFB-BFHZ)
Ask authors/readers for more resources
Motivation: In statistical bioinformatics research, different optimization mechanisms potentially lead to 'over-optimism' in published papers. So far, however, a systematic critical study concerning the various sources underlying this over-optimism is lacking. Results: We present an empirical study on over-optimism using high-dimensional classification as example. Specifically, we consider a 'promising' new classification algorithm, namely linear discriminant analysis incorporating prior knowledge on gene functional groups through an appropriate shrinkage of the within-group covariance matrix. While this approach yields poor results in terms of error rate, we quantitatively demonstrate that it can artificially seem superior to existing approaches if we 'fish for significance'. The investigated sources of over-optimism include the optimization of datasets, of settings, of competing methods and, most importantly, of the method's characteristics. We conclude that, if the improvement of a quantitative criterion such as the error rate is the main contribution of a paper, the superiority of new algorithms should always be demonstrated on independent validation data.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available