Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses
<|reference_start|>Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses: Rebuses are puzzles requiring constrained multi-step reasoning to identify a hidden phrase from a set of images and letters. In this work, we introduce a large collection of verbalized rebuses for the Italian language and use it to assess the rebus-solving capabilities of state-of-the-art large language models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly on this task, ad-hoc fine-tuning seems to improve models' performance. However, we find that performance gains from training are largely motivated by memorization. Our results suggest that rebus solving remains a challenging test bed to evaluate large language models' linguistic proficiency and sequential instruction-following skills.<|reference_end|>
@article{sarti2024non,
    title={Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian
  Rebuses},
    author={Gabriele Sarti, Tommaso Caselli, Malvina Nissim, Arianna Bisazza},
    journal={arXiv preprint arXiv:2408.00584},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.00584},
    primaryClass={cs.CL cs.AI}
}