title: Futga: Towards Fine-grained Music Understanding through
    Temporally-enhanced Generative Augmentation
summary: Existing music captioning methods are limited to generating concise global
  descriptions of short music clips, which fail to capture fine-grained musical
  characteristics and time-aware musical changes. To address these limitations,
  we propose FUTGA, a model equipped with fined-grained music understanding
  capabilities through learning from generative augmentation with temporal
  compositions. We leverage existing music caption datasets and large language
  models (LLMs) to synthesize fine-grained music captions with structural
  descriptions and time boundaries for full-length songs. Augmented by the
  proposed synthetic dataset, FUTGA is enabled to identify the music's temporal
  changes at key transition points and their musical functions, as well as
  generate detailed descriptions for each music segment. We further introduce a
  full-length music caption dataset generated by FUTGA, as the augmentation of
  the MusicCaps and the Song Describer datasets. We evaluate the automatically
  generated captions on several downstream tasks, including music generation and
  retrieval. The experiments demonstrate the quality of the generated captions
  and the better performance in various downstream tasks achieved by the proposed
  music captioning approach. Our code and datasets can be found in
  https://huggingface.co/JoshuaW1997/FUTGA{blue{https://huggingface.co/JoshuaW1997/FUTGA}}.
paper_authors._id: 66a9ae6f35070a5a52a2419e
paper_authors.name: Junda Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-31T08:26:14.965Z
paper_authors._id: 616c0cd640e2f69baa1c7adf
paper_authors.avatarUrl: /avatars/9ad6ada73b3fb2d5fb696ec91f8edb1b.svg
paper_authors.fullname: Junda Wu
paper_authors.type: user
paper_authors.user: JoshuaW1997
paper_authors._id: 66a9ae6f35070a5a52a2419f
paper_authors.name: Zachary Novack
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-31T07:44:04.364Z
paper_authors._id: 643060c6cb3fe707b24c53a2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643060c6cb3fe707b24c53a2/MIoM9hrX0vV4XRyrm-4Kz.jpeg
paper_authors.fullname: Zachary Novack
paper_authors.type: user
paper_authors.user: ZacharyNovack
paper_authors._id: 66a9ae6f35070a5a52a241a0
paper_authors.name: Amit Namburi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-31T08:26:22.399Z
paper_authors._id: 6611fb5555f5219b33a35856
paper_authors.avatarUrl: /avatars/4ed9acc339d15acd8a5ce821acdac025.svg
paper_authors.fullname: Amit Namburi
paper_authors.type: user
paper_authors.user: namburiamit
paper_authors._id: 66a9ae6f35070a5a52a241a1
paper_authors.name: Jiaheng Dai
paper_authors._id: 66a9ae6f35070a5a52a241a2
paper_authors.name: Hao-Wen Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-31T08:26:52.860Z
paper_authors._id: 649cb9456209253a76854798
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649cb9456209253a76854798/OY2_nPMMxkds2ME2j9xte.jpeg
paper_authors.fullname: Hao-Wen Dong
paper_authors.type: user
paper_authors.user: salu133445
paper_authors._id: 66a9ae6f35070a5a52a241a3
paper_authors.name: Zhouhang Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-31T08:27:04.736Z
paper_authors._id: 6187199bb1085ab638324e82
paper_authors.avatarUrl: /avatars/02281e4e5dc260fff62de34589f42b7d.svg
paper_authors.fullname: zhouhang xie
paper_authors.type: user
paper_authors.user: zhohanx
paper_authors._id: 66a9ae6f35070a5a52a241a4
paper_authors.name: Carol Chen
paper_authors._id: 66a9ae6f35070a5a52a241a5
paper_authors.name: Julian McAuley
paper_title: Futga: Towards Fine-grained Music Understanding through
    Temporally-enhanced Generative Augmentation
paper_summary: Existing music captioning methods are limited to generating concise global
  descriptions of short music clips, which fail to capture fine-grained musical
  characteristics and time-aware musical changes. To address these limitations,
  we propose FUTGA, a model equipped with fined-grained music understanding
  capabilities through learning from generative augmentation with temporal
  compositions. We leverage existing music caption datasets and large language
  models (LLMs) to synthesize fine-grained music captions with structural
  descriptions and time boundaries for full-length songs. Augmented by the
  proposed synthetic dataset, FUTGA is enabled to identify the music's temporal
  changes at key transition points and their musical functions, as well as
  generate detailed descriptions for each music segment. We further introduce a
  full-length music caption dataset generated by FUTGA, as the augmentation of
  the MusicCaps and the Song Describer datasets. We evaluate the automatically
  generated captions on several downstream tasks, including music generation and
  retrieval. The experiments demonstrate the quality of the generated captions
  and the better performance in various downstream tasks achieved by the proposed
  music captioning approach. Our code and datasets can be found in
  https://huggingface.co/JoshuaW1997/FUTGA{blue{https://huggingface.co/JoshuaW1997/FUTGA}}.
paper_ai_summary: FUTGA enhances music captioning by generating detailed, temporally-aware descriptions using generative augmentation with temporal compositions and large language models.
paper_ai_keywords: fined-grained music understanding generative augmentation temporal compositions full-length music captions structural descriptions time boundaries key transition points musical functions music caption dataset MusicCaps Song Describer music generation music retrieval