SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes
<|reference_start|>SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes: This paper presents the results of the SHROOM, a shared task focused on detecting hallucinations: outputs from natural language generation (NLG) systems that are fluent, yet inaccurate. Such cases of overgeneration put in jeopardy many NLG applications, where correctness is often mission-critical. The shared task was conducted with a newly constructed dataset of 4000 model outputs labeled by 5 annotators each, spanning 3 NLP tasks: machine translation, paraphrase generation and definition modeling.   The shared task was tackled by a total of 58 different users grouped in 42 teams, out of which 27 elected to write a system description paper; collectively, they submitted over 300 prediction sets on both tracks of the shared task. We observe a number of key trends in how this approach was tackled -- many participants rely on a handful of model, and often rely either on synthetic data for fine-tuning or zero-shot prompting strategies. While a majority of the teams did outperform our proposed baseline system, the performances of top-scoring systems are still consistent with a random handling of the more challenging items.<|reference_end|>
@article{mickus2024semeval-2024,
    title={SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and
  Related Observable Overgeneration Mistakes},
    author={Timothee Mickus, Elaine Zosa, Ra'ul V'azquez, Teemu Vahtola, J"org
  Tiedemann, Vincent Segonne, Alessandro Raganato, Marianna Apidianaki},
    journal={arXiv preprint arXiv:2403.07726},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.07726},
    primaryClass={cs.CL}
}