4.5 Article

Emerging Trends: SOTA-Chasing

期刊

NATURAL LANGUAGE ENGINEERING
卷 28, 期 2, 页码 249-269

出版社

CAMBRIDGE UNIV PRESS
DOI: 10.1017/S1351324922000043

关键词

State-of-the-art; Evaluation; Benchmarks; Leaderboards; Root causes; Leadership; Reviewing; Replication crisis

向作者/读者索取更多资源

Pursuing state-of-the-art (SOTA) numbers in research papers can have costs, such as missing out on more promising opportunities and potentially leading to unrealistic expectations. Lack of leadership and uncertain reviewing processes are identified as the root causes of SOTA-chasing. This phenomenon is compared to the replication crisis in scientific literature.
Many papers are chasing state-of-the-art (SOTA) numbers, and more will do so in the future. SOTA-chasing comes with many costs. SOTA-chasing squeezes out more promising opportunities such as coopetition and interdisciplinary collaboration. In addition, there is a risk that too much SOTA-chasing could lead to claims of superhuman performance, unrealistic expectations, and the next AI winter. Two root causes for SOTA-chasing will be discussed: (1) lack of leadership and (2) iffy reviewing processes. SOTA-chasing may be similar to the replication crisis in the scientific literature. The replication crisis is yet another example, like evaluation, of over-confidence in accepted practices and the scientific method, even when such practices lead to absurd consequences.

作者

我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。

评论

主要评分

4.5
评分不足

次要评分

新颖性
-
重要性
-
科学严谨性
-
评价这篇论文

推荐

暂无数据
暂无数据