3.8 Proceedings Paper

Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks

Related references

Note: Only part of the references are listed.
Article Computer Science, Theory & Methods

A Survey on Bias and Fairness in Machine Learning

Ninareh Mehrabi et al.

Summary: With the widespread use of AI systems in everyday life, fairness in design has become crucial. Researchers have developed methods to address biases in different subdomains and established a taxonomy for fairness definitions. Existing work shows biases in AI applications, and researchers are working on solutions to mitigate these problems.

ACM COMPUTING SURVEYS (2021)

Proceedings Paper Computer Science, Artificial Intelligence

Aggregating Complex Annotations via Merging and Matching

Alexander Braylan et al.

Summary: This study addresses the challenge of resolving annotator disagreement in complex annotation tasks and proposes a general aggregation model to support label merging across diverse tasks. Additionally, for multi-object annotation tasks that require multiple labels per item, the model provides a method to match annotator labels to entities to ensure only matching labels are aggregated.

KDD '21: PROCEEDINGS OF THE 27TH ACM SIGKDD CONFERENCE ON KNOWLEDGE DISCOVERY & DATA MINING (2021)

Proceedings Paper Computer Science, Information Systems

Crowdsourced Text Sequence Aggregation based on Hybrid Reliability and Representation

Jiyi Li

PROCEEDINGS OF THE 43RD INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL (SIGIR '20) (2020)

Proceedings Paper Computer Science, Information Systems

OpenCrowd: A Human-Al Collaborative Approach for Finding Social Influencers via Open-Ended Answers Aggregation

Ines Arous et al.

WEB CONFERENCE 2020: PROCEEDINGS OF THE WORLD WIDE WEB CONFERENCE (WWW 2020) (2020)

Proceedings Paper Computer Science, Information Systems

Modeling and Aggregation of Complex Annotations via Annotation Distances

Alexander Braylan et al.

WEB CONFERENCE 2020: PROCEEDINGS OF THE WORLD WIDE WEB CONFERENCE (WWW 2020) (2020)

Proceedings Paper Computer Science, Artificial Intelligence

Lean Crowdsourcing: Combining Humans and Machines in an Online System

Steve Branson et al.

30TH IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2017) (2017)

Article Computer Science, Artificial Intelligence

The Unified and Holistic Method Gamma (γ) for Inter-Annotator Agreement Measure and Alignment

Yann Mathet et al.

COMPUTATIONAL LINGUISTICS (2015)

Article Computer Science, Artificial Intelligence

The Unified and Holistic Method Gamma (γ) for Inter-Annotator Agreement Measure and Alignment

Yann Mathet et al.

COMPUTATIONAL LINGUISTICS (2015)

Review Computer Science, Artificial Intelligence

Inter-Coder Agreement for Computational Linguistics

Ron Artstein et al.

COMPUTATIONAL LINGUISTICS (2008)