e-space
Manchester Metropolitan University's Research Repository

    xiacui at SemEval-2023 Task 11: Learning a Model in Mixed-Annotator Datasets Using Annotator Ranking Scores as Training Weights

    Cui, Xia ORCID logoORCID: https://orcid.org/0000-0002-1726-3814 (2023) xiacui at SemEval-2023 Task 11: Learning a Model in Mixed-Annotator Datasets Using Annotator Ranking Scores as Training Weights. In: The 17th International Workshop on Semantic Evaluation (SemEval-2023), 13 July 2023 - 14 July 2023, Toronto, Canada.

    [img]
    Preview
    Published Version
    Available under License Creative Commons Attribution.

    Download (213kB) | Preview

    Abstract

    This paper describes the development of a system for SemEval-2023 Shared Task 11 on Learning with Disagreements (Le-WiDi) (Leonardelllietal.,2023). Labelled data plays a vital role in the development of machine learning systems. The human-annotated labels are usually considered the truth for training or validation. To obtain truth labels, a traditional way is to hire domain experts to perform an expensive annotation process. Crowd-sourcing labelling is comparably cheap, whereas it raises a question on the reliability of annotators. A common strategy in a mixed-annotator dataset with various sets of annotators for each instance is to aggregate the labels among multiple groups of annotators to obtain the truth labels. However, these annotators might not reach an agreement, and there is no guarantee of the reliability of these labels either. With further problems caused by human label variation, subjective tasks usually suffer from the different opinions provided by the annotators. In this paper, we propose two simple heuristic functions to compute the annotator ranking scores, namely AnnoHard and AnnoSoft, based on the hard labels (i.e., aggregative labels) and soft labels (i.e., cross-entropy values). By introducing these scores, we adjust the weights of the training instances to improve the learning with disagreements among the annotators.

    Impact and Reach

    Statistics

    Activity Overview
    6 month trend
    46Downloads
    6 month trend
    51Hits

    Additional statistics for this dataset are available via IRStats2.

    Repository staff only

    Edit record Edit record