title: WavJourney: Compositional Audio Creation with Large Language Models
summary: Large Language Models (LLMs) have shown great promise in integrating diverse
  expert models to tackle intricate language and vision tasks. Despite their
  significance in advancing the field of Artificial Intelligence Generated
  Content (AIGC), their potential in intelligent audio content creation remains
  unexplored. In this work, we tackle the problem of creating audio content with
  storylines encompassing speech, music, and sound effects, guided by text
  instructions. We present WavJourney, a system that leverages LLMs to connect
  various audio models for audio content generation. Given a text description of
  an auditory scene, WavJourney first prompts LLMs to generate a structured
  script dedicated to audio storytelling. The audio script incorporates diverse
  audio elements, organized based on their spatio-temporal relationships. As a
  conceptual representation of audio, the audio script provides an interactive
  and interpretable rationale for human engagement. Afterward, the audio script
  is fed into a script compiler, converting it into a computer program. Each line
  of the program calls a task-specific audio generation model or computational
  operation function (e.g., concatenate, mix). The computer program is then
  executed to obtain an explainable solution for audio generation. We demonstrate
  the practicality of WavJourney across diverse real-world scenarios, including
  science fiction, education, and radio play. The explainable and interactive
  design of WavJourney fosters human-machine co-creation in multi-round
  dialogues, enhancing creative control and adaptability in audio production.
  WavJourney audiolizes the human imagination, opening up new avenues for
  creativity in multimedia content creation.
paper_authors._id: 64c1c7a27179ef53b87db51a
paper_authors.name: Xubo Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-27T12:28:48.980Z
paper_authors._id: 64c260105f9e98008d498081
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c260105f9e98008d498081/TwJKuPtGMn9WL7T-54WGg.jpeg
paper_authors.fullname: Xubo Liu
paper_authors.type: user
paper_authors.user: Xubo-Liu
paper_authors._id: 64c1c7a27179ef53b87db51b
paper_authors.name: Zhongkai Zhu
paper_authors._id: 64c1c7a27179ef53b87db51c
paper_authors.name: Haohe Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T11:46:44.647Z
paper_authors._id: 6155245d1c762d4d61b51d5d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
paper_authors.fullname: haoheliu
paper_authors.type: user
paper_authors.user: haoheliu
paper_authors._id: 64c1c7a27179ef53b87db51d
paper_authors.name: Yi Yuan
paper_authors._id: 64c1c7a27179ef53b87db51e
paper_authors.name: Meng Cui
paper_authors._id: 64c1c7a27179ef53b87db51f
paper_authors.name: Qiushi Huang
paper_authors._id: 64c1c7a27179ef53b87db520
paper_authors.name: Jinhua Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T11:48:13.786Z
paper_authors._id: 6462416c8bd7025ebe821d33
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6462416c8bd7025ebe821d33/SmXxd_ZcQCWISLwzAZiLS.jpeg
paper_authors.fullname: Jinhua Liang
paper_authors.type: user
paper_authors.user: JinhuaL1ANG
paper_authors._id: 64c1c7a27179ef53b87db521
paper_authors.name: Yin Cao
paper_authors._id: 64c1c7a27179ef53b87db522
paper_authors.name: Qiuqiang Kong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T11:49:13.295Z
paper_authors._id: 64311ee3288c97756736a5ca
paper_authors.avatarUrl: /avatars/e6dc01880149450ebc2b9a07c19920cc.svg
paper_authors.fullname: Qiuqiang Kong
paper_authors.type: user
paper_authors.user: qiuqiangkong
paper_authors._id: 64c1c7a27179ef53b87db523
paper_authors.name: Mark D. Plumbley
paper_authors._id: 64c1c7a27179ef53b87db524
paper_authors.name: Wenwu Wang
paper_title: WavJourney: Compositional Audio Creation with Large Language Models
paper_summary: Large Language Models (LLMs) have shown great promise in integrating diverse
  expert models to tackle intricate language and vision tasks. Despite their
  significance in advancing the field of Artificial Intelligence Generated
  Content (AIGC), their potential in intelligent audio content creation remains
  unexplored. In this work, we tackle the problem of creating audio content with
  storylines encompassing speech, music, and sound effects, guided by text
  instructions. We present WavJourney, a system that leverages LLMs to connect
  various audio models for audio content generation. Given a text description of
  an auditory scene, WavJourney first prompts LLMs to generate a structured
  script dedicated to audio storytelling. The audio script incorporates diverse
  audio elements, organized based on their spatio-temporal relationships. As a
  conceptual representation of audio, the audio script provides an interactive
  and interpretable rationale for human engagement. Afterward, the audio script
  is fed into a script compiler, converting it into a computer program. Each line
  of the program calls a task-specific audio generation model or computational
  operation function (e.g., concatenate, mix). The computer program is then
  executed to obtain an explainable solution for audio generation. We demonstrate
  the practicality of WavJourney across diverse real-world scenarios, including
  science fiction, education, and radio play. The explainable and interactive
  design of WavJourney fosters human-machine co-creation in multi-round
  dialogues, enhancing creative control and adaptability in audio production.
  WavJourney audiolizes the human imagination, opening up new avenues for
  creativity in multimedia content creation.
paper_ai_keywords: Large Language Models (LLMs) Artificial Intelligence Generated Content (AIGC) audio content generation structured script audio script audio models spatio-temporal relationships script compiler computer program task-specific audio generation models human-machine co-creation