A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models
<|reference_start|>A Hard Nut to Crack: Idiom Detection with Conversational Large Language Models: In this work, we explore idiomatic language processing with Large Language Models (LLMs). We introduce the Idiomatic language Test Suite IdioTS, a new dataset of difficult examples specifically designed by language experts to assess the capabilities of LLMs to process figurative language at sentence level. We propose a comprehensive evaluation methodology based on an idiom detection task, where LLMs are prompted with detecting an idiomatic expression in a given English sentence. We present a thorough automatic and manual evaluation of the results and an extensive error analysis.<|reference_end|>
@article{fornaciari2024a,
    title={A Hard Nut to Crack: Idiom Detection with Conversational Large Language
  Models},
    author={Francesca De Luca Fornaciari, Bego~na Altuna, Itziar Gonzalez-Dios,
  Maite Melero},
    journal={arXiv preprint arXiv:2405.10579},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.10579},
    primaryClass={cs.CL}
}