title: IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task
    Language Understanding
summary: Known by more than 1.5 billion people in the Indian subcontinent, Indic
  languages present unique challenges and opportunities for natural language
  processing (NLP) research due to their rich cultural heritage, linguistic
  diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark
  designed to evaluate Large Language Models (LLMs) across Indic languages,
  building upon the MMLU Pro (Massive Multitask Language Understanding)
  framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi,
  Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique
  challenges and opportunities presented by the linguistic diversity of the
  Indian subcontinent. This benchmark encompasses a wide range of tasks in
  language comprehension, reasoning, and generation, meticulously crafted to
  capture the intricacies of Indian languages. IndicMMLU-Pro provides a
  standardized evaluation framework to push the research boundaries in Indic
  language AI, facilitating the development of more accurate, efficient, and
  culturally sensitive models. This paper outlines the benchmarks' design
  principles, task taxonomy, and data collection methodology, and presents
  baseline results from state-of-the-art multilingual models.
paper_authors._id: 6799946c18cb282841d42639
paper_authors.name: Sankalp KJ
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:27:40.565Z
paper_authors._id: 664e218ea10258b399d5c358
paper_authors.avatarUrl: /avatars/77f2ee0949560504e1643263fd7084da.svg
paper_authors.fullname: Sankalp KJ
paper_authors.type: user
paper_authors.user: SankalpKJ
paper_authors._id: 6799946c18cb282841d4263a
paper_authors.name: Ashutosh Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T21:06:22.798Z
paper_authors._id: 65c7fa3a242e3ee0c656927d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c7fa3a242e3ee0c656927d/yulc-eFl2hntzfbRJWiox.jpeg
paper_authors.fullname: Ashutosh Kumar
paper_authors.type: user
paper_authors.user: ashu-1069
paper_authors._id: 6799946c18cb282841d4263b
paper_authors.name: Laxmaan Balaji
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-29T02:37:34.240Z
paper_authors._id: 66707b60405252abeefd4c50
paper_authors.avatarUrl: /avatars/ee2728f115376e234e96820b8b376849.svg
paper_authors.fullname: Laxmaan Balaji
paper_authors.type: user
paper_authors.user: laxmaanb
paper_authors._id: 6799946c18cb282841d4263c
paper_authors.name: Nikunj Kotecha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:28:09.074Z
paper_authors._id: 66c607f85eae54107879aee9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c607f85eae54107879aee9/m2rRKcN9H6baABGzN-Fzr.png
paper_authors.fullname: Nikunj Kotecha
paper_authors.type: user
paper_authors.user: nikunjkotecha
paper_authors._id: 6799946c18cb282841d4263d
paper_authors.name: Vinija Jain
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:28:16.742Z
paper_authors._id: 6517bfecce732bf33a29d04b
paper_authors.avatarUrl: /avatars/b6534a540fa10199df8f9acc497083d5.svg
paper_authors.fullname: Vinija Jain
paper_authors.type: user
paper_authors.user: Vinija
paper_authors._id: 6799946c18cb282841d4263e
paper_authors.name: Aman Chadha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T08:55:14.312Z
paper_authors._id: 63a4754927f1f64ed7238dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
paper_authors.fullname: Aman Chadha
paper_authors.type: user
paper_authors.user: amanchadha
paper_authors._id: 6799946c18cb282841d4263f
paper_authors.name: Sreyoshi Bhaduri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:28:22.871Z
paper_authors._id: 6760b279deb31f2c1e5b4f42
paper_authors.avatarUrl: /avatars/b3863ab0d63d052198dc9b4261def623.svg
paper_authors.fullname: sreyoshi bhaduri
paper_authors.type: user
paper_authors.user: sreyoshibhaduri
paper_title: IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task
    Language Understanding
paper_summary: Known by more than 1.5 billion people in the Indian subcontinent, Indic
  languages present unique challenges and opportunities for natural language
  processing (NLP) research due to their rich cultural heritage, linguistic
  diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark
  designed to evaluate Large Language Models (LLMs) across Indic languages,
  building upon the MMLU Pro (Massive Multitask Language Understanding)
  framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi,
  Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique
  challenges and opportunities presented by the linguistic diversity of the
  Indian subcontinent. This benchmark encompasses a wide range of tasks in
  language comprehension, reasoning, and generation, meticulously crafted to
  capture the intricacies of Indian languages. IndicMMLU-Pro provides a
  standardized evaluation framework to push the research boundaries in Indic
  language AI, facilitating the development of more accurate, efficient, and
  culturally sensitive models. This paper outlines the benchmarks' design
  principles, task taxonomy, and data collection methodology, and presents
  baseline results from state-of-the-art multilingual models.
paper_ai_keywords: Large Language Models (LLMs) MMLU Pro (Massive Multitask Language Understanding) language comprehension reasoning generation IndicMMLU-Pro