期刊
NATURAL LANGUAGE ENGINEERING
卷 28, 期 2, 页码 249-269出版社
CAMBRIDGE UNIV PRESS
DOI: 10.1017/S1351324922000043
关键词
State-of-the-art; Evaluation; Benchmarks; Leaderboards; Root causes; Leadership; Reviewing; Replication crisis
Pursuing state-of-the-art (SOTA) numbers in research papers can have costs, such as missing out on more promising opportunities and potentially leading to unrealistic expectations. Lack of leadership and uncertain reviewing processes are identified as the root causes of SOTA-chasing. This phenomenon is compared to the replication crisis in scientific literature.
Many papers are chasing state-of-the-art (SOTA) numbers, and more will do so in the future. SOTA-chasing comes with many costs. SOTA-chasing squeezes out more promising opportunities such as coopetition and interdisciplinary collaboration. In addition, there is a risk that too much SOTA-chasing could lead to claims of superhuman performance, unrealistic expectations, and the next AI winter. Two root causes for SOTA-chasing will be discussed: (1) lack of leadership and (2) iffy reviewing processes. SOTA-chasing may be similar to the replication crisis in the scientific literature. The replication crisis is yet another example, like evaluation, of over-confidence in accepted practices and the scientific method, even when such practices lead to absurd consequences.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据