The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models
<|reference_start|>The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models: Benchmark plays a pivotal role in assessing the advancements of large language models (LLMs). While numerous benchmarks have been proposed to evaluate LLMs' capabilities, there is a notable absence of a dedicated benchmark for assessing their musical abilities. To address this gap, we present ZIQI-Eval, a comprehensive and large-scale music benchmark specifically designed to evaluate the music-related capabilities of LLMs. ZIQI-Eval encompasses a wide range of questions, covering 10 major categories and 56 subcategories, resulting in over 14,000 meticulously curated data entries. By leveraging ZIQI-Eval, we conduct a comprehensive evaluation over 16 LLMs to evaluate and analyze LLMs' performance in the domain of music. Results indicate that all LLMs perform poorly on the ZIQI-Eval benchmark, suggesting significant room for improvement in their musical capabilities. With ZIQI-Eval, we aim to provide a standardized and robust evaluation framework that facilitates a comprehensive assessment of LLMs' music-related abilities. The dataset is available at GitHub\footnote{https://github.com/zcli-charlie/ZIQI-Eval} and HuggingFace\footnote{https://huggingface.co/datasets/MYTH-Lab/ZIQI-Eval}.<|reference_end|>
@article{li2024the,
    title={The Music Maestro or The Musically Challenged, A Massive Music
  Evaluation Benchmark for Large Language Models},
    author={Jiajia Li, Lu Yang, Mingni Tang, Cong Chen, Zuchao Li, Ping Wang, Hai
  Zhao},
    journal={arXiv preprint arXiv:2406.15885},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.15885},
    primaryClass={cs.SD cs.AI eess.AS}
}