title: IberBench: LLM Evaluation on Iberian Languages
summary: Large Language Models (LLMs) remain difficult to evaluate comprehensively,
  particularly for languages other than English, where high-quality data is often
  limited. Existing benchmarks and leaderboards are predominantly
  English-centric, with only a few addressing other languages. These benchmarks
  fall short in several key areas: they overlook the diversity of language
  varieties, prioritize fundamental Natural Language Processing (NLP)
  capabilities over tasks of industrial relevance, and are static. With these
  aspects in mind, we present IberBench, a comprehensive and extensible benchmark
  designed to assess LLM performance on both fundamental and industry-relevant
  NLP tasks, in languages spoken across the Iberian Peninsula and Ibero-America.
  IberBench integrates 101 datasets from evaluation campaigns and recent
  benchmarks, covering 22 task categories such as sentiment and emotion analysis,
  toxicity detection, and summarization. The benchmark addresses key limitations
  in current evaluation practices, such as the lack of linguistic diversity and
  static evaluation setups by enabling continual updates and community-driven
  model and dataset submissions moderated by a committee of experts. We evaluate
  23 LLMs ranging from 100 million to 14 billion parameters and provide empirical
  insights into their strengths and limitations. Our findings indicate that (i)
  LLMs perform worse on industry-relevant tasks than in fundamental ones, (ii)
  performance is on average lower for Galician and Basque, (iii) some tasks show
  results close to random, and (iv) in other tasks LLMs perform above random but
  below shared task systems. IberBench offers open-source implementations for the
  entire evaluation pipeline, including dataset normalization and hosting,
  incremental evaluation of LLMs, and a publicly accessible leaderboard.
paper_authors._id: 680b40774d69b6950c4eabda
paper_authors.name: José Ángel González
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-25T12:27:15.281Z
paper_authors._id: 613f7e43e9cc36f11c7715b1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631558612397-613f7e43e9cc36f11c7715b1.png
paper_authors.fullname: José Ángel González
paper_authors.type: user
paper_authors.user: jogonba2
paper_authors._id: 680b40774d69b6950c4eabdb
paper_authors.name: Ian Borrego Obrador
paper_authors._id: 680b40774d69b6950c4eabdc
paper_authors.name: Álvaro Romo Herrero
paper_authors._id: 680b40774d69b6950c4eabdd
paper_authors.name: Areg Mikael Sarvazyan
paper_authors._id: 680b40774d69b6950c4eabde
paper_authors.name: Mara Chinea-Ríos
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-25T07:58:32.606Z
paper_authors._id: 60f95c8fda0985b973d59d77
paper_authors.avatarUrl: /avatars/5606f0191b9f86e6b55f7e5ab6cc8bb6.svg
paper_authors.fullname: Mara Chinea Rios
paper_authors.type: user
paper_authors.user: mchinea
paper_authors._id: 680b40774d69b6950c4eabdf
paper_authors.name: Angelo Basile
paper_authors._id: 680b40774d69b6950c4eabe0
paper_authors.name: Marc Franco-Salvador
paper_title: IberBench: LLM Evaluation on Iberian Languages
paper_summary: Large Language Models (LLMs) remain difficult to evaluate comprehensively,
  particularly for languages other than English, where high-quality data is often
  limited. Existing benchmarks and leaderboards are predominantly
  English-centric, with only a few addressing other languages. These benchmarks
  fall short in several key areas: they overlook the diversity of language
  varieties, prioritize fundamental Natural Language Processing (NLP)
  capabilities over tasks of industrial relevance, and are static. With these
  aspects in mind, we present IberBench, a comprehensive and extensible benchmark
  designed to assess LLM performance on both fundamental and industry-relevant
  NLP tasks, in languages spoken across the Iberian Peninsula and Ibero-America.
  IberBench integrates 101 datasets from evaluation campaigns and recent
  benchmarks, covering 22 task categories such as sentiment and emotion analysis,
  toxicity detection, and summarization. The benchmark addresses key limitations
  in current evaluation practices, such as the lack of linguistic diversity and
  static evaluation setups by enabling continual updates and community-driven
  model and dataset submissions moderated by a committee of experts. We evaluate
  23 LLMs ranging from 100 million to 14 billion parameters and provide empirical
  insights into their strengths and limitations. Our findings indicate that (i)
  LLMs perform worse on industry-relevant tasks than in fundamental ones, (ii)
  performance is on average lower for Galician and Basque, (iii) some tasks show
  results close to random, and (iv) in other tasks LLMs perform above random but
  below shared task systems. IberBench offers open-source implementations for the
  entire evaluation pipeline, including dataset normalization and hosting,
  incremental evaluation of LLMs, and a publicly accessible leaderboard.
paper_ai_summary: IberBench is a comprehensive benchmark for evaluating Large Language Models on diverse tasks across languages spoken in the Iberian Peninsula and Ibero-America, addressing limitations of current evaluation practices.
paper_ai_keywords: Large Language Models LLMs IberBench Natural Language Processing NLP sentiment and emotion analysis toxicity detection summarization linguistic diversity dataset normalization