期刊
IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING
卷 32, 期 8, 页码 1586-1594出版社
IEEE COMPUTER SOC
DOI: 10.1109/TKDE.2019.2912815
关键词
Dependency relationship; k-fold cross validation; reliable estimate; replication; variance
类别
资金
- Ministry of Science and Technology in Taiwan [106-2410-H-006-020]
It is popular to evaluate the performance of classification algorithms by k-fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform k-fold cross validation. Most of them did not consider the correlation among the replications of k-fold cross validation, and hence the variance could be underestimated. The purpose of this study is to explore whether k-fold cross validation should be repeatedly performed for obtaining reliable accuracy estimates. The dependency relationships between the predictions of the same instance in two replications of k-fold cross validation are first analyzed for k-nearest neighbors with k = 1. Then, statistical methods are proposed to test the strength of the dependency level between the accuracy estimates resulting from two replications of k-fold cross validation. The experimental results on 20 data sets show that the accuracy estimates obtained from various replications of k-fold cross validation are generally highly correlated, and the correlation will be higher as the number of folds increases. The k-fold cross validation with a large number of folds and a small number of replications should be adopted for performance evaluation of classification algorithms.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据