title: MEGAVERSE: Benchmarking Large Language Models Across Languages,
    Modalities, Models and Tasks
summary: Recently, there has been a rapid advancement in research on Large Language
  Models (LLMs), resulting in significant progress in several Natural Language
  Processing (NLP) tasks. Consequently, there has been a surge in LLM evaluation
  research to comprehend the models' capabilities and limitations. However, much
  of this research has been confined to the English language, leaving LLM
  building and evaluation for non-English languages relatively unexplored. There
  has been an introduction of several new LLMs, necessitating their evaluation on
  non-English languages. This study aims to expand our MEGA benchmarking suite by
  including six new datasets to form the MEGAVERSE benchmark. The benchmark
  comprises 22 datasets covering 81 languages, including low-resource African
  languages. We evaluate several state-of-the-art LLMs like GPT-3.5-Turbo, GPT4,
  PaLM2, and Llama2 on the MEGAVERSE datasets. Additionally, we include two
  multimodal datasets in the benchmark and assess the performance of the
  LLaVa-v1.5 model. Our experiments suggest that GPT4 and PaLM2 outperform the
  Llama models on various tasks, notably on low-resource languages, with GPT4
  outperforming PaLM2 on more datasets than vice versa. However, issues such as
  data contamination must be addressed to obtain an accurate assessment of LLM
  performance on non-English languages.
paper_authors._id: 6552efbdf2a8f0b4c6cc761c
paper_authors.name: Sanchit Ahuja
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T11:00:38.820Z
paper_authors._id: 64e5d1cca9a5eabaa6fff445
paper_authors.avatarUrl: /avatars/e492fbfc10111454a83ac9ffb8cf72f1.svg
paper_authors.fullname: Sanchit Ahuja
paper_authors.type: user
paper_authors.user: sahuja
paper_authors._id: 6552efbdf2a8f0b4c6cc761d
paper_authors.name: Divyanshu Aggarwal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T11:01:58.728Z
paper_authors._id: 60f44e08a13e78c866903053
paper_authors.avatarUrl: /avatars/475ecc9df9f2c4d96d3dde392b70a9ca.svg
paper_authors.fullname: Divyanshu Aggarwal
paper_authors.type: user
paper_authors.user: Divyanshu
paper_authors._id: 6552efbdf2a8f0b4c6cc761e
paper_authors.name: Varun Gumma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T11:01:15.122Z
paper_authors._id: 64e5f244abf028d943a38350
paper_authors.avatarUrl: /avatars/52ac934a28e354f5962e8a8b02ec887a.svg
paper_authors.fullname: Varun Gumma
paper_authors.type: user
paper_authors.user: VarunGumma
paper_authors._id: 6552efbdf2a8f0b4c6cc761f
paper_authors.name: Ishaan Watts
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T11:01:49.454Z
paper_authors._id: 6487e8a4756b9f10338dbb25
paper_authors.avatarUrl: /avatars/5b3967cbfd801d4a50834998b508819e.svg
paper_authors.fullname: Ishaan Watts
paper_authors.type: user
paper_authors.user: WattsIshaan
paper_authors._id: 6552efbdf2a8f0b4c6cc7620
paper_authors.name: Ashutosh Sathe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T11:02:07.260Z
paper_authors._id: 639b1ec77a0c15a95bbf911b
paper_authors.avatarUrl: /avatars/5dc5406b096733d733cba29ce42d2ceb.svg
paper_authors.fullname: Ashutosh Sathe
paper_authors.type: user
paper_authors.user: ashutoshbsathe
paper_authors._id: 6552efbdf2a8f0b4c6cc7621
paper_authors.name: Millicent Ochieng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T11:02:13.916Z
paper_authors._id: 612c76001dba3c21765b911d
paper_authors.avatarUrl: /avatars/6b11f09f3f6a3bdedde3980e3ca92ce8.svg
paper_authors.fullname: Millicent Ochieng
paper_authors.type: user
paper_authors.user: millicentochieng
paper_authors._id: 6552efbdf2a8f0b4c6cc7622
paper_authors.name: Rishav Hada
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T11:02:20.890Z
paper_authors._id: 605229ff72705624f02496e2
paper_authors.avatarUrl: /avatars/3a3353c8a509ddfc1096d1eb6d5ab576.svg
paper_authors.fullname: Rishav Hada
paper_authors.type: user
paper_authors.user: hadarishav
paper_authors._id: 6552efbdf2a8f0b4c6cc7623
paper_authors.name: Prachi Jain
paper_authors._id: 6552efbdf2a8f0b4c6cc7624
paper_authors.name: Maxamed Axmed
paper_authors._id: 6552efbdf2a8f0b4c6cc7625
paper_authors.name: Kalika Bali
paper_authors._id: 6552efbdf2a8f0b4c6cc7626
paper_authors.name: Sunayana Sitaram
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-11-14T03:56:40.109Z
paper_authors._id: 63edb3b7684767daecb04417
paper_authors.avatarUrl: /avatars/9265619db1a1bd3c2470ef22e0d95dc4.svg
paper_authors.fullname: Sunayana Sitaram
paper_authors.type: user
paper_authors.user: Sunayana
paper_title: MEGAVERSE: Benchmarking Large Language Models Across Languages,
    Modalities, Models and Tasks
paper_summary: Recently, there has been a rapid advancement in research on Large Language
  Models (LLMs), resulting in significant progress in several Natural Language
  Processing (NLP) tasks. Consequently, there has been a surge in LLM evaluation
  research to comprehend the models' capabilities and limitations. However, much
  of this research has been confined to the English language, leaving LLM
  building and evaluation for non-English languages relatively unexplored. There
  has been an introduction of several new LLMs, necessitating their evaluation on
  non-English languages. This study aims to expand our MEGA benchmarking suite by
  including six new datasets to form the MEGAVERSE benchmark. The benchmark
  comprises 22 datasets covering 81 languages, including low-resource African
  languages. We evaluate several state-of-the-art LLMs like GPT-3.5-Turbo, GPT4,
  PaLM2, and Llama2 on the MEGAVERSE datasets. Additionally, we include two
  multimodal datasets in the benchmark and assess the performance of the
  LLaVa-v1.5 model. Our experiments suggest that GPT4 and PaLM2 outperform the
  Llama models on various tasks, notably on low-resource languages, with GPT4
  outperforming PaLM2 on more datasets than vice versa. However, issues such as
  data contamination must be addressed to obtain an accurate assessment of LLM
  performance on non-English languages.
paper_ai_summary: A study expands the MEGA benchmarking suite to MEGAVERSE with 22 datasets covering 81 languages, evaluating various LLMs including GPT-3.5-Turbo, GPT4, PaLM2, Llama2, and LLaVa-v1.5, highlighting performance gaps between models on low-resource languages.
paper_ai_keywords: Large Language Models LLMs Natural Language Processing NLP MEGA benchmark MEGAVERSE GPT-3.5-Turbo GPT4 PaLM2 Llama2 LLaVa-v1.5 low-resource languages data contamination