KMMLU: Measuring Massive Multitask Language Understanding in Korean
<|reference_start|>KMMLU: Measuring Massive Multitask Language Understanding in Korean: We propose KMMLU, a new Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects ranging from humanities to STEM. While prior Korean benchmarks are translated from existing English benchmarks, KMMLU is collected from original Korean exams, capturing linguistic and cultural aspects of the Korean language. We test 27 public and proprietary LLMs and observe the best public model to score 50.5%, leaving significant room for improvement. This model was primarily trained for English and Chinese, not Korean. Current LLMs tailored to Korean, such as Polyglot-Ko, perform far worse. Surprisingly, even the most capable proprietary LLMs, e.g., GPT-4 and HyperCLOVA X do not exceed 60%. This suggests that further work is needed to improve LLMs for Korean, and we believe KMMLU offers the appropriate tool to track this progress. We make our dataset publicly available on the Hugging Face Hub and integrate the benchmark into EleutherAI's Language Model Evaluation Harness.<|reference_end|>
@article{son2024kmmlu:,
    title={KMMLU: Measuring Massive Multitask Language Understanding in Korean},
    author={Guijin Son and Hanwool Lee and Sungdong Kim and Seungone Kim and
  Niklas Muennighoff and Taekyoon Choi and Cheonbok Park and Kang Min Yoo and
  Stella Biderman},
    journal={arXiv preprint arXiv:2402.11548},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.11548},
    primaryClass={cs.CL}
}