MEGAVERSE: Benchmarking Large Language Models Across Languages,
  Modalities, Models and Tasks
Recently, there has been a rapid advancement in research on Large Language
Models (LLMs), resulting in significant progress in several Natural Language
Processing (NLP) tasks. Consequently, there has been a surge in LLM evaluation
research to comprehend the models' capabilities and limitations. However, much
of this research has been confined to the English language, leaving LLM
building and evaluation for non-English languages relatively unexplored. There
has been an introduction of several new LLMs, necessitating their evaluation on
non-English languages. This study aims to expand our MEGA benchmarking suite by
including six new datasets to form the MEGAVERSE benchmark. The benchmark
comprises 22 datasets covering 81 languages, including low-resource African
languages. We evaluate several state-of-the-art LLMs like GPT-3.5-Turbo, GPT4,
PaLM2, and Llama2 on the MEGAVERSE datasets. Additionally, we include two
multimodal datasets in the benchmark and assess the performance of the
LLaVa-v1.5 model. Our experiments suggest that GPT4 and PaLM2 outperform the
Llama models on various tasks, notably on low-resource languages, with GPT4
outperforming PaLM2 on more datasets than vice versa. However, issues such as
data contamination must be addressed to obtain an accurate assessment of LLM
performance on non-English languages.
6552efbdf2a8f0b4c6cc761c
Sanchit Ahuja
admin_assigned
2023-11-14T11:00:38.820Z
64e5d1cca9a5eabaa6fff445
/avatars/e492fbfc10111454a83ac9ffb8cf72f1.svg
Sanchit Ahuja
user
sahuja
6552efbdf2a8f0b4c6cc761d
Divyanshu Aggarwal
admin_assigned
2023-11-14T11:01:58.728Z
60f44e08a13e78c866903053
/avatars/475ecc9df9f2c4d96d3dde392b70a9ca.svg
Divyanshu Aggarwal
user
Divyanshu
6552efbdf2a8f0b4c6cc761e
Varun Gumma
admin_assigned
2023-11-14T11:01:15.122Z
64e5f244abf028d943a38350
/avatars/52ac934a28e354f5962e8a8b02ec887a.svg
Varun Gumma
user
VarunGumma
6552efbdf2a8f0b4c6cc761f
Ishaan Watts
admin_assigned
2023-11-14T11:01:49.454Z
6487e8a4756b9f10338dbb25
/avatars/5b3967cbfd801d4a50834998b508819e.svg
Ishaan Watts
user
WattsIshaan
6552efbdf2a8f0b4c6cc7620
Ashutosh Sathe
admin_assigned
2023-11-14T11:02:07.260Z
639b1ec77a0c15a95bbf911b
/avatars/5dc5406b096733d733cba29ce42d2ceb.svg
Ashutosh Sathe
user
ashutoshbsathe
6552efbdf2a8f0b4c6cc7621
Millicent Ochieng
admin_assigned
2023-11-14T11:02:13.916Z
612c76001dba3c21765b911d
/avatars/6b11f09f3f6a3bdedde3980e3ca92ce8.svg
Millicent Ochieng
user
millicentochieng
6552efbdf2a8f0b4c6cc7622
Rishav Hada
admin_assigned
2023-11-14T11:02:20.890Z
605229ff72705624f02496e2
/avatars/3a3353c8a509ddfc1096d1eb6d5ab576.svg
Rishav Hada
user
hadarishav
6552efbdf2a8f0b4c6cc7623
Prachi Jain
6552efbdf2a8f0b4c6cc7624
Maxamed Axmed
6552efbdf2a8f0b4c6cc7625
Kalika Bali
6552efbdf2a8f0b4c6cc7626
Sunayana Sitaram
extracted_confirmed
2023-11-14T03:56:40.109Z
63edb3b7684767daecb04417
/avatars/9265619db1a1bd3c2470ef22e0d95dc4.svg
Sunayana Sitaram
user
Sunayana
MEGAVERSE: Benchmarking Large Language Models Across Languages,
  Modalities, Models and Tasks
Recently, there has been a rapid advancement in research on Large Language
Models (LLMs), resulting in significant progress in several Natural Language
Processing (NLP) tasks. Consequently, there has been a surge in LLM evaluation
research to comprehend the models' capabilities and limitations. However, much
of this research has been confined to the English language, leaving LLM
building and evaluation for non-English languages relatively unexplored. There
has been an introduction of several new LLMs, necessitating their evaluation on
non-English languages. This study aims to expand our MEGA benchmarking suite by
including six new datasets to form the MEGAVERSE benchmark. The benchmark
comprises 22 datasets covering 81 languages, including low-resource African
languages. We evaluate several state-of-the-art LLMs like GPT-3.5-Turbo, GPT4,
PaLM2, and Llama2 on the MEGAVERSE datasets. Additionally, we include two
multimodal datasets in the benchmark and assess the performance of the
LLaVa-v1.5 model. Our experiments suggest that GPT4 and PaLM2 outperform the
Llama models on various tasks, notably on low-resource languages, with GPT4
outperforming PaLM2 on more datasets than vice versa. However, issues such as
data contamination must be addressed to obtain an accurate assessment of LLM
performance on non-English languages.
A study expands the MEGA benchmarking suite to MEGAVERSE with 22 datasets covering 81 languages, evaluating various LLMs including GPT-3.5-Turbo, GPT4, PaLM2, Llama2, and LLaVa-v1.5, highlighting performance gaps between models on low-resource languages.
Large Language Models LLMs Natural Language Processing NLP MEGA benchmark MEGAVERSE GPT-3.5-Turbo GPT4 PaLM2 Llama2 LLaVa-v1.5 low-resource languages data contamination