Eka-Eval : A Comprehensive Evaluation Framework for Large Language
  Models in Indian Languages
The rapid advancement of Large Language Models (LLMs) has intensified the
need for evaluation frameworks that go beyond English centric benchmarks and
address the requirements of linguistically diverse regions such as India. We
present EKA-EVAL, a unified and production-ready evaluation framework that
integrates over 35 benchmarks, including 10 Indic-specific datasets, spanning
categories like reasoning, mathematics, tool use, long-context understanding,
and reading comprehension. Compared to existing Indian language evaluation
tools, EKA-EVAL offers broader benchmark coverage, with built-in support for
distributed inference, quantization, and multi-GPU usage. Our systematic
comparison positions EKA-EVAL as the first end-to-end, extensible evaluation
suite tailored for both global and Indic LLMs, significantly lowering the
barrier to multilingual benchmarking. The framework is open-source and publicly
available at https://github.com/lingo-iitgn/ eka-eval and a part of ongoing EKA
initiative (https://eka.soket.ai), which aims to scale up to over 100
benchmarks and establish a robust, multilingual evaluation ecosystem for LLMs.
686b4e69213f123a1f88bd76
Samridhi Raj Sinha
claimed_verified
2025-07-08T12:40:30.588Z
665454167473f3c257f89e0f
/avatars/c48bc493a9e0248a828cb899b77a7efb.svg
Samridhi Raj Sinha
user
samiee2213
686b4e69213f123a1f88bd77
Rajvee Sheth
claimed_verified
2025-07-07T11:24:11.116Z
66e1425c919f283fbd7dfb5e
https://cdn-avatars.huggingface.co/v1/production/uploads/66e1425c919f283fbd7dfb5e/91Cumv-y20I6x0xHDOP9z.jpeg
Rajvee Sheth
user
RajveeSheth
686b4e69213f123a1f88bd78
Abhishek Upperwal
686b4e69213f123a1f88bd79
Mayank Singh
Eka-Eval : A Comprehensive Evaluation Framework for Large Language
  Models in Indian Languages
The rapid advancement of Large Language Models (LLMs) has intensified the
need for evaluation frameworks that go beyond English centric benchmarks and
address the requirements of linguistically diverse regions such as India. We
present EKA-EVAL, a unified and production-ready evaluation framework that
integrates over 35 benchmarks, including 10 Indic-specific datasets, spanning
categories like reasoning, mathematics, tool use, long-context understanding,
and reading comprehension. Compared to existing Indian language evaluation
tools, EKA-EVAL offers broader benchmark coverage, with built-in support for
distributed inference, quantization, and multi-GPU usage. Our systematic
comparison positions EKA-EVAL as the first end-to-end, extensible evaluation
suite tailored for both global and Indic LLMs, significantly lowering the
barrier to multilingual benchmarking. The framework is open-source and publicly
available at https://github.com/lingo-iitgn/ eka-eval and a part of ongoing EKA
initiative (https://eka.soket.ai), which aims to scale up to over 100
benchmarks and establish a robust, multilingual evaluation ecosystem for LLMs.
EKA-EVAL is a comprehensive evaluation framework for Large Language Models that includes diverse benchmarks, supports distributed inference, and is tailored for both global and Indic languages.
Large Language Models EKA-EVAL Indic-specific datasets reasoning mathematics tool use long-context understanding reading comprehension distributed inference quantization multi-GPU usage end-to-end extensible evaluation suite multilingual benchmarking