title: Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian
    Rebuses
summary: Rebuses are puzzles requiring constrained multi-step reasoning to identify a
  hidden phrase from a set of images and letters. In this work, we introduce a
  large collection of verbalized rebuses for the Italian language and use it to
  assess the rebus-solving capabilities of state-of-the-art large language
  models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly
  on this task, ad-hoc fine-tuning seems to improve models' performance. However,
  we find that performance gains from training are largely motivated by
  memorization. Our results suggest that rebus solving remains a challenging test
  bed to evaluate large language models' linguistic proficiency and sequential
  instruction-following skills.
paper_authors._id: 66ac7b62c019f71636813668
paper_authors.name: Gabriele Sarti
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-08-02T06:29:32.764Z
paper_authors._id: 5e7749883d77a72421292d07
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670231290373-5e7749883d77a72421292d07.jpeg
paper_authors.fullname: Gabriele Sarti
paper_authors.type: user
paper_authors.user: gsarti
paper_authors._id: 66ac7b62c019f71636813669
paper_authors.name: Tommaso Caselli
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-02T06:23:31.564Z
paper_authors._id: 60cde5cd4061635e43b28f44
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60cde5cd4061635e43b28f44/f9JlUdh39gfoCnESqMtic.jpeg
paper_authors.fullname: Tommaso Caselli
paper_authors.type: user
paper_authors.user: tommasoc
paper_authors._id: 66ac7b62c019f7163681366a
paper_authors.name: Malvina Nissim
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-02T06:23:31.564Z
paper_authors._id: 6246f49c0569474a9623e05a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648817396108-6246f49c0569474a9623e05a.jpeg
paper_authors.fullname: Malvina Nissim
paper_authors.type: user
paper_authors.user: mnissim
paper_authors._id: 66ac7b62c019f7163681366b
paper_authors.name: Arianna Bisazza
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T13:33:08.481Z
paper_authors._id: 61e050c73a1781f66b4e9ab6
paper_authors.avatarUrl: /avatars/9c91a18cdc53587422311fd13a14833e.svg
paper_authors.fullname: Arianna Bisazza
paper_authors.type: user
paper_authors.user: arianna-bis
paper_title: Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian
    Rebuses
paper_summary: Rebuses are puzzles requiring constrained multi-step reasoning to identify a
  hidden phrase from a set of images and letters. In this work, we introduce a
  large collection of verbalized rebuses for the Italian language and use it to
  assess the rebus-solving capabilities of state-of-the-art large language
  models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly
  on this task, ad-hoc fine-tuning seems to improve models' performance. However,
  we find that performance gains from training are largely motivated by
  memorization. Our results suggest that rebus solving remains a challenging test
  bed to evaluate large language models' linguistic proficiency and sequential
  instruction-following skills.