MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language
  Models
Multimodal models that jointly process audio and language hold great promise
in audio understanding and are increasingly being adopted in the music domain.
By allowing users to query via text and obtain information about a given audio
input, these models have the potential to enable a variety of music
understanding tasks via language-based interfaces. However, their evaluation
poses considerable challenges, and it remains unclear how to effectively assess
their ability to correctly interpret music-related inputs with current methods.
Motivated by this, we introduce MuChoMusic, a benchmark for evaluating music
understanding in multimodal language models focused on audio. MuChoMusic
comprises 1,187 multiple-choice questions, all validated by human annotators,
on 644 music tracks sourced from two publicly available music datasets, and
covering a wide variety of genres. Questions in the benchmark are crafted to
assess knowledge and reasoning abilities across several dimensions that cover
fundamental musical concepts and their relation to cultural and functional
contexts. Through the holistic analysis afforded by the benchmark, we evaluate
five open-source models and identify several pitfalls, including an
over-reliance on the language modality, pointing to a need for better
multimodal integration. Data and code are open-sourced.
66b04458674a309496934f0a
Benno Weck
admin_assigned
2024-08-05T09:37:16.824Z
656a0e9b79a834846b75fc0a
https://cdn-avatars.huggingface.co/v1/production/uploads/656a0e9b79a834846b75fc0a/JI54JseoCnJB3NAnUHFI1.png
Benno Weck
user
bennoweck
66b04458674a309496934f0b
Ilaria Manco
admin_assigned
2024-08-05T09:37:22.640Z
64ed060f33d50752497d4a16
https://cdn-avatars.huggingface.co/v1/production/uploads/64ed060f33d50752497d4a16/0QbK9jeGvY3DBMVHDdTDX.jpeg
Ilaria Manco
user
ilaria-manco
66b04458674a309496934f0c
Emmanouil Benetos
claimed_verified
2024-08-05T10:24:05.273Z
66b0a0839797b94f001ed874
https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
Emmanouil Benetos
user
emmanouilb
66b04458674a309496934f0d
Elio Quinton
admin_assigned
2024-08-05T09:37:33.367Z
64f1c4bf4fc14420f6935e7f
/avatars/b01e80f10d0de3f6b0da8b617299d9b7.svg
Elio Quinton
user
elio-quinton-umg
66b04458674a309496934f0e
George Fazekas
66b04458674a309496934f0f
Dmitry Bogdanov
admin_assigned
2024-08-05T09:38:21.429Z
656776c7d0a121b8e8e67a16
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/UBWZOXWV2am6a_NHsuwi5.png
Dmitry Bogdanov
user
dbogdanov
MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language
  Models
Multimodal models that jointly process audio and language hold great promise
in audio understanding and are increasingly being adopted in the music domain.
By allowing users to query via text and obtain information about a given audio
input, these models have the potential to enable a variety of music
understanding tasks via language-based interfaces. However, their evaluation
poses considerable challenges, and it remains unclear how to effectively assess
their ability to correctly interpret music-related inputs with current methods.
Motivated by this, we introduce MuChoMusic, a benchmark for evaluating music
understanding in multimodal language models focused on audio. MuChoMusic
comprises 1,187 multiple-choice questions, all validated by human annotators,
on 644 music tracks sourced from two publicly available music datasets, and
covering a wide variety of genres. Questions in the benchmark are crafted to
assess knowledge and reasoning abilities across several dimensions that cover
fundamental musical concepts and their relation to cultural and functional
contexts. Through the holistic analysis afforded by the benchmark, we evaluate
five open-source models and identify several pitfalls, including an
over-reliance on the language modality, pointing to a need for better
multimodal integration. Data and code are open-sourced.
MuChoMusic is a benchmark that evaluates multimodal language models focusing on audio understanding through multiple-choice questions covering various musical and contextual dimensions.
multimodal models audio understanding language-based interfaces benchmark multiple-choice questions human annotators music tracks knowledge assessment reasoning abilities fundamental musical concepts cultural contexts functional contexts multimodal integration