title: MusicAgent: An AI Agent for Music Understanding and Generation with
    Large Language Models
summary: AI-empowered music processing is a diverse field that encompasses dozens of
  tasks, ranging from generation tasks (e.g., timbre synthesis) to comprehension
  tasks (e.g., music classification). For developers and amateurs, it is very
  difficult to grasp all of these task to satisfy their requirements in music
  processing, especially considering the huge differences in the representations
  of music data and the model applicability across platforms among various tasks.
  Consequently, it is necessary to build a system to organize and integrate these
  tasks, and thus help practitioners to automatically analyze their demand and
  call suitable tools as solutions to fulfill their requirements. Inspired by the
  recent success of large language models (LLMs) in task automation, we develop a
  system, named MusicAgent, which integrates numerous music-related tools and an
  autonomous workflow to address user requirements. More specifically, we build
  1) toolset that collects tools from diverse sources, including Hugging Face,
  GitHub, and Web API, etc. 2) an autonomous workflow empowered by LLMs (e.g.,
  ChatGPT) to organize these tools and automatically decompose user requests into
  multiple sub-tasks and invoke corresponding music tools. The primary goal of
  this system is to free users from the intricacies of AI-music tools, enabling
  them to concentrate on the creative aspect. By granting users the freedom to
  effortlessly combine tools, the system offers a seamless and enriching music
  experience.
paper_authors._id: 6530bfea6072a8d5c7bf5adf
paper_authors.name: Dingyao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-19T08:20:25.594Z
paper_authors._id: 63182ae5743def9b36173285
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/CnSkXSQ_S3Qf9xZssOEBV.png
paper_authors.fullname: Dingyao Yu
paper_authors.type: user
paper_authors.user: Uranus
paper_authors._id: 6530bfea6072a8d5c7bf5ae0
paper_authors.name: Kaitao Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-19T08:21:01.211Z
paper_authors._id: 5fc0b2b61160c47d1d438568
paper_authors.avatarUrl: /avatars/b355912b0ec683e73f21c8d36620e146.svg
paper_authors.fullname: Kaitao Song
paper_authors.type: user
paper_authors.user: KaitaoSong
paper_authors._id: 6530bfea6072a8d5c7bf5ae1
paper_authors.name: Peiling Lu
paper_authors._id: 6530bfea6072a8d5c7bf5ae2
paper_authors.name: Tianyu He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-19T09:41:08.924Z
paper_authors._id: 619b7b1cab4c7b7f16a7d59e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/619b7b1cab4c7b7f16a7d59e/6TvXaAqBghAMYO1-j5l4v.jpeg
paper_authors.fullname: Tianyu He
paper_authors.type: user
paper_authors.user: deeptimhe
paper_authors._id: 6530bfea6072a8d5c7bf5ae3
paper_authors.name: Xu Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-19T08:22:19.425Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 6530bfea6072a8d5c7bf5ae4
paper_authors.name: Wei Ye
paper_authors._id: 6530bfea6072a8d5c7bf5ae5
paper_authors.name: Shikun Zhang
paper_authors._id: 6530bfea6072a8d5c7bf5ae6
paper_authors.name: Jiang Bian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-19T08:23:01.485Z
paper_authors._id: 63f253f8f4e30ffd2bd308fb
paper_authors.avatarUrl: /avatars/303f4c7ee588f638acf78a7966786e1e.svg
paper_authors.fullname: Jiang Bian
paper_authors.type: user
paper_authors.user: bianjiang
paper_title: MusicAgent: An AI Agent for Music Understanding and Generation with
    Large Language Models
paper_summary: AI-empowered music processing is a diverse field that encompasses dozens of
  tasks, ranging from generation tasks (e.g., timbre synthesis) to comprehension
  tasks (e.g., music classification). For developers and amateurs, it is very
  difficult to grasp all of these task to satisfy their requirements in music
  processing, especially considering the huge differences in the representations
  of music data and the model applicability across platforms among various tasks.
  Consequently, it is necessary to build a system to organize and integrate these
  tasks, and thus help practitioners to automatically analyze their demand and
  call suitable tools as solutions to fulfill their requirements. Inspired by the
  recent success of large language models (LLMs) in task automation, we develop a
  system, named MusicAgent, which integrates numerous music-related tools and an
  autonomous workflow to address user requirements. More specifically, we build
  1) toolset that collects tools from diverse sources, including Hugging Face,
  GitHub, and Web API, etc. 2) an autonomous workflow empowered by LLMs (e.g.,
  ChatGPT) to organize these tools and automatically decompose user requests into
  multiple sub-tasks and invoke corresponding music tools. The primary goal of
  this system is to free users from the intricacies of AI-music tools, enabling
  them to concentrate on the creative aspect. By granting users the freedom to
  effortlessly combine tools, the system offers a seamless and enriching music
  experience.
paper_ai_summary: MusicAgent is a system that integrates various music-related tools and uses large language models to automate music processing tasks based on user requests.
paper_ai_keywords: large language models LLMs ChatGPT autonomous workflow timbre synthesis music classification task automation