期刊
INFORMATION RETRIEVAL
卷 12, 期 1, 页码 81-97出版社
SPRINGER
DOI: 10.1007/s10791-008-9072-x
关键词
Reference standards; Evaluation; Inter-annotator agreement; Text mining; Information retrieval
With the help of a team of expert biologist judges, the TREC Genomics track has generated four large sets of gold standard'' test collections, comprised of over a hundred unique topics, two kinds of ad hoc retrieval tasks, and their corresponding relevance judgments. Over the years of the track, increasingly complex tasks necessitated the creation of judging tools and training guidelines to accommodate teams of part-time short-term workers from a variety of specialized biological scientific backgrounds, and to address consistency and reproducibility of the assessment process. Important lessons were learned about factors that influenced the utility of the test collections including topic design, annotations provided by judges, methods used for identifying and training judges, and providing a central moderator meta-judge''.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据