IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task
  Language Understanding
Known by more than 1.5 billion people in the Indian subcontinent, Indic
languages present unique challenges and opportunities for natural language
processing (NLP) research due to their rich cultural heritage, linguistic
diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark
designed to evaluate Large Language Models (LLMs) across Indic languages,
building upon the MMLU Pro (Massive Multitask Language Understanding)
framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi,
Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique
challenges and opportunities presented by the linguistic diversity of the
Indian subcontinent. This benchmark encompasses a wide range of tasks in
language comprehension, reasoning, and generation, meticulously crafted to
capture the intricacies of Indian languages. IndicMMLU-Pro provides a
standardized evaluation framework to push the research boundaries in Indic
language AI, facilitating the development of more accurate, efficient, and
culturally sensitive models. This paper outlines the benchmarks' design
principles, task taxonomy, and data collection methodology, and presents
baseline results from state-of-the-art multilingual models.
6799946c18cb282841d42639
Sankalp KJ
admin_assigned
2025-01-29T17:27:40.565Z
664e218ea10258b399d5c358
/avatars/77f2ee0949560504e1643263fd7084da.svg
Sankalp KJ
user
SankalpKJ
6799946c18cb282841d4263a
Ashutosh Kumar
claimed_verified
2025-01-29T21:06:22.798Z
65c7fa3a242e3ee0c656927d
https://cdn-avatars.huggingface.co/v1/production/uploads/65c7fa3a242e3ee0c656927d/yulc-eFl2hntzfbRJWiox.jpeg
Ashutosh Kumar
user
ashu-1069
6799946c18cb282841d4263b
Laxmaan Balaji
extracted_pending
2025-01-29T02:37:34.240Z
66707b60405252abeefd4c50
/avatars/ee2728f115376e234e96820b8b376849.svg
Laxmaan Balaji
user
laxmaanb
6799946c18cb282841d4263c
Nikunj Kotecha
admin_assigned
2025-01-29T17:28:09.074Z
66c607f85eae54107879aee9
https://cdn-avatars.huggingface.co/v1/production/uploads/66c607f85eae54107879aee9/m2rRKcN9H6baABGzN-Fzr.png
Nikunj Kotecha
user
nikunjkotecha
6799946c18cb282841d4263d
Vinija Jain
admin_assigned
2025-01-29T17:28:16.742Z
6517bfecce732bf33a29d04b
/avatars/b6534a540fa10199df8f9acc497083d5.svg
Vinija Jain
user
Vinija
6799946c18cb282841d4263e
Aman Chadha
claimed_verified
2025-01-29T08:55:14.312Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
6799946c18cb282841d4263f
Sreyoshi Bhaduri
admin_assigned
2025-01-29T17:28:22.871Z
6760b279deb31f2c1e5b4f42
/avatars/b3863ab0d63d052198dc9b4261def623.svg
sreyoshi bhaduri
user
sreyoshibhaduri
IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task
  Language Understanding
Known by more than 1.5 billion people in the Indian subcontinent, Indic
languages present unique challenges and opportunities for natural language
processing (NLP) research due to their rich cultural heritage, linguistic
diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark
designed to evaluate Large Language Models (LLMs) across Indic languages,
building upon the MMLU Pro (Massive Multitask Language Understanding)
framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi,
Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique
challenges and opportunities presented by the linguistic diversity of the
Indian subcontinent. This benchmark encompasses a wide range of tasks in
language comprehension, reasoning, and generation, meticulously crafted to
capture the intricacies of Indian languages. IndicMMLU-Pro provides a
standardized evaluation framework to push the research boundaries in Indic
language AI, facilitating the development of more accurate, efficient, and
culturally sensitive models. This paper outlines the benchmarks' design
principles, task taxonomy, and data collection methodology, and presents
baseline results from state-of-the-art multilingual models.
IndicMMLU-Pro evaluates multilingual large language models across various Indic languages using a comprehensive benchmark that covers comprehension, reasoning, and generation tasks.
Large Language Models MMLU Pro language comprehension reasoning generation Indic languages