期刊
BEHAVIOR RESEARCH METHODS
卷 -, 期 -, 页码 -出版社
SPRINGER
DOI: 10.3758/s13428-023-02246-7
关键词
Aberrant responding; Bots; Outlier detection; Permutation test; Unsupervised learning
This article introduces a model-agnostic and unsupervised bot detection algorithm that uses permutation test with leave-one-out calculations of outlier statistics. The algorithm provides a p value for each respondent to determine if they are a bot. The simulation study shows that the proposed algorithm outperforms naive alternatives in terms of sensitivity calibration and classification accuracy.
To detect bots in online survey data, there is a wealth of literature on statistical detection using only responses to Likert-type items. There are two traditions in the literature. One tradition requires labeled data, forgoing strong model assumptions. The other tradition requires a measurement model, forgoing collection of labeled data. In the present article, we consider the problem where neither requirement is available, for an inventory that has the same number of Likert-type categories for all items. We propose a bot detection algorithm that is both model-agnostic and unsupervised. Our proposed algorithm involves a permutation test with leave-one-out calculations of outlier statistics. For each respondent, it outputs a p value for the null hypothesis that the respondent is a bot. Such an algorithm offers nominal sensitivity calibration that is robust to the bot response distribution. In a simulation study, we found our proposed algorithm to improve upon naive alternatives in terms of 95% sensitivity calibration and, in many scenarios, in terms of classification accuracy.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据