title: MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language
    Models
summary: Multimodal models that jointly process audio and language hold great promise
  in audio understanding and are increasingly being adopted in the music domain.
  By allowing users to query via text and obtain information about a given audio
  input, these models have the potential to enable a variety of music
  understanding tasks via language-based interfaces. However, their evaluation
  poses considerable challenges, and it remains unclear how to effectively assess
  their ability to correctly interpret music-related inputs with current methods.
  Motivated by this, we introduce MuChoMusic, a benchmark for evaluating music
  understanding in multimodal language models focused on audio. MuChoMusic
  comprises 1,187 multiple-choice questions, all validated by human annotators,
  on 644 music tracks sourced from two publicly available music datasets, and
  covering a wide variety of genres. Questions in the benchmark are crafted to
  assess knowledge and reasoning abilities across several dimensions that cover
  fundamental musical concepts and their relation to cultural and functional
  contexts. Through the holistic analysis afforded by the benchmark, we evaluate
  five open-source models and identify several pitfalls, including an
  over-reliance on the language modality, pointing to a need for better
  multimodal integration. Data and code are open-sourced.
paper_authors._id: 66b04458674a309496934f0a
paper_authors.name: Benno Weck
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T09:37:16.824Z
paper_authors._id: 656a0e9b79a834846b75fc0a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656a0e9b79a834846b75fc0a/JI54JseoCnJB3NAnUHFI1.png
paper_authors.fullname: Benno Weck
paper_authors.type: user
paper_authors.user: bennoweck
paper_authors._id: 66b04458674a309496934f0b
paper_authors.name: Ilaria Manco
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T09:37:22.640Z
paper_authors._id: 64ed060f33d50752497d4a16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ed060f33d50752497d4a16/0QbK9jeGvY3DBMVHDdTDX.jpeg
paper_authors.fullname: Ilaria Manco
paper_authors.type: user
paper_authors.user: ilaria-manco
paper_authors._id: 66b04458674a309496934f0c
paper_authors.name: Emmanouil Benetos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-05T10:24:05.273Z
paper_authors._id: 66b0a0839797b94f001ed874
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
paper_authors.fullname: Emmanouil Benetos
paper_authors.type: user
paper_authors.user: emmanouilb
paper_authors._id: 66b04458674a309496934f0d
paper_authors.name: Elio Quinton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T09:37:33.367Z
paper_authors._id: 64f1c4bf4fc14420f6935e7f
paper_authors.avatarUrl: /avatars/b01e80f10d0de3f6b0da8b617299d9b7.svg
paper_authors.fullname: Elio Quinton
paper_authors.type: user
paper_authors.user: elio-quinton-umg
paper_authors._id: 66b04458674a309496934f0e
paper_authors.name: George Fazekas
paper_authors._id: 66b04458674a309496934f0f
paper_authors.name: Dmitry Bogdanov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T09:38:21.429Z
paper_authors._id: 656776c7d0a121b8e8e67a16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/UBWZOXWV2am6a_NHsuwi5.png
paper_authors.fullname: Dmitry Bogdanov
paper_authors.type: user
paper_authors.user: dbogdanov
paper_title: MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language
    Models
paper_summary: Multimodal models that jointly process audio and language hold great promise
  in audio understanding and are increasingly being adopted in the music domain.
  By allowing users to query via text and obtain information about a given audio
  input, these models have the potential to enable a variety of music
  understanding tasks via language-based interfaces. However, their evaluation
  poses considerable challenges, and it remains unclear how to effectively assess
  their ability to correctly interpret music-related inputs with current methods.
  Motivated by this, we introduce MuChoMusic, a benchmark for evaluating music
  understanding in multimodal language models focused on audio. MuChoMusic
  comprises 1,187 multiple-choice questions, all validated by human annotators,
  on 644 music tracks sourced from two publicly available music datasets, and
  covering a wide variety of genres. Questions in the benchmark are crafted to
  assess knowledge and reasoning abilities across several dimensions that cover
  fundamental musical concepts and their relation to cultural and functional
  contexts. Through the holistic analysis afforded by the benchmark, we evaluate
  five open-source models and identify several pitfalls, including an
  over-reliance on the language modality, pointing to a need for better
  multimodal integration. Data and code are open-sourced.
paper_ai_summary: MuChoMusic is a benchmark that evaluates multimodal language models focusing on audio understanding through multiple-choice questions covering various musical and contextual dimensions.
paper_ai_keywords: multimodal models audio understanding language-based interfaces benchmark multiple-choice questions human annotators music tracks knowledge assessment reasoning abilities fundamental musical concepts cultural contexts functional contexts multimodal integration