WavJourney: Compositional Audio Creation with Large Language Models
Large Language Models (LLMs) have shown great promise in integrating diverse
expert models to tackle intricate language and vision tasks. Despite their
significance in advancing the field of Artificial Intelligence Generated
Content (AIGC), their potential in intelligent audio content creation remains
unexplored. In this work, we tackle the problem of creating audio content with
storylines encompassing speech, music, and sound effects, guided by text
instructions. We present WavJourney, a system that leverages LLMs to connect
various audio models for audio content generation. Given a text description of
an auditory scene, WavJourney first prompts LLMs to generate a structured
script dedicated to audio storytelling. The audio script incorporates diverse
audio elements, organized based on their spatio-temporal relationships. As a
conceptual representation of audio, the audio script provides an interactive
and interpretable rationale for human engagement. Afterward, the audio script
is fed into a script compiler, converting it into a computer program. Each line
of the program calls a task-specific audio generation model or computational
operation function (e.g., concatenate, mix). The computer program is then
executed to obtain an explainable solution for audio generation. We demonstrate
the practicality of WavJourney across diverse real-world scenarios, including
science fiction, education, and radio play. The explainable and interactive
design of WavJourney fosters human-machine co-creation in multi-round
dialogues, enhancing creative control and adaptability in audio production.
WavJourney audiolizes the human imagination, opening up new avenues for
creativity in multimedia content creation.
64c1c7a27179ef53b87db51a
Xubo Liu
claimed_verified
2023-07-27T12:28:48.980Z
64c260105f9e98008d498081
https://cdn-avatars.huggingface.co/v1/production/uploads/64c260105f9e98008d498081/TwJKuPtGMn9WL7T-54WGg.jpeg
Xubo Liu
user
Xubo-Liu
64c1c7a27179ef53b87db51b
Zhongkai Zhu
64c1c7a27179ef53b87db51c
Haohe Liu
admin_assigned
2023-07-27T11:46:44.647Z
6155245d1c762d4d61b51d5d
https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
haoheliu
user
haoheliu
64c1c7a27179ef53b87db51d
Yi Yuan
64c1c7a27179ef53b87db51e
Meng Cui
64c1c7a27179ef53b87db51f
Qiushi Huang
64c1c7a27179ef53b87db520
Jinhua Liang
admin_assigned
2023-07-27T11:48:13.786Z
6462416c8bd7025ebe821d33
https://cdn-avatars.huggingface.co/v1/production/uploads/6462416c8bd7025ebe821d33/SmXxd_ZcQCWISLwzAZiLS.jpeg
Jinhua Liang
user
JinhuaL1ANG
64c1c7a27179ef53b87db521
Yin Cao
64c1c7a27179ef53b87db522
Qiuqiang Kong
admin_assigned
2023-07-27T11:49:13.295Z
64311ee3288c97756736a5ca
/avatars/e6dc01880149450ebc2b9a07c19920cc.svg
Qiuqiang Kong
user
qiuqiangkong
64c1c7a27179ef53b87db523
Mark D. Plumbley
64c1c7a27179ef53b87db524
Wenwu Wang
WavJourney: Compositional Audio Creation with Large Language Models
Large Language Models (LLMs) have shown great promise in integrating diverse
expert models to tackle intricate language and vision tasks. Despite their
significance in advancing the field of Artificial Intelligence Generated
Content (AIGC), their potential in intelligent audio content creation remains
unexplored. In this work, we tackle the problem of creating audio content with
storylines encompassing speech, music, and sound effects, guided by text
instructions. We present WavJourney, a system that leverages LLMs to connect
various audio models for audio content generation. Given a text description of
an auditory scene, WavJourney first prompts LLMs to generate a structured
script dedicated to audio storytelling. The audio script incorporates diverse
audio elements, organized based on their spatio-temporal relationships. As a
conceptual representation of audio, the audio script provides an interactive
and interpretable rationale for human engagement. Afterward, the audio script
is fed into a script compiler, converting it into a computer program. Each line
of the program calls a task-specific audio generation model or computational
operation function (e.g., concatenate, mix). The computer program is then
executed to obtain an explainable solution for audio generation. We demonstrate
the practicality of WavJourney across diverse real-world scenarios, including
science fiction, education, and radio play. The explainable and interactive
design of WavJourney fosters human-machine co-creation in multi-round
dialogues, enhancing creative control and adaptability in audio production.
WavJourney audiolizes the human imagination, opening up new avenues for
creativity in multimedia content creation.
WavJourney leverages Large Language Models to generate structured audio scripts from text, which are then processed by task-specific audio models to create audio content, enabling explainable and human-controlled audio production.
Large Language Models WavJourney audio storytelling audio scripts spatio-temporal relationships script compiler audio generation models computational operation functions audio content generation interactive rationale human-machine co-creation