SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detection
<|reference_start|>SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based Classification for Hallucination Detection: We describe the University of Amsterdam Intelligent Data Engineering Lab team's entry for the SemEval-2024 Task 6 competition. The SHROOM-INDElab system builds on previous work on using prompt programming and in-context learning with large language models (LLMs) to build classifiers for hallucination detection, and extends that work through the incorporation of context-specific definition of task, role, and target concept, and automated generation of examples for use in a few-shot prompting approach. The resulting system achieved fourth-best and sixth-best performance in the model-agnostic track and model-aware tracks for Task 6, respectively, and evaluation using the validation sets showed that the system's classification decisions were consistent with those of the crowd-sourced human labellers. We further found that a zero-shot approach provided better accuracy than a few-shot approach using automatically generated examples. Code for the system described in this paper is available on Github.<|reference_end|>
@article{allen2024shroom-indelab,
    title={SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based
  Classification for Hallucination Detection},
    author={Bradley P. Allen and Fina Polat and Paul Groth},
    journal={arXiv preprint arXiv:2404.03732},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.03732},
    primaryClass={cs.CL cs.AI}
}