V2Meow: Meowing to the Visual Beat via Music Generation
Generating high quality music that complements the visual content of a video
is a challenging task. Most existing visual conditioned music generation
systems generate symbolic music data, such as MIDI files, instead of raw audio
waveform. Given the limited availability of symbolic music data, such methods
can only generate music for a few instruments or for specific types of visual
input. In this paper, we propose a novel approach called V2Meow that can
generate high-quality music audio that aligns well with the visual semantics of
a diverse range of video input types. Specifically, the proposed music
generation system is a multi-stage autoregressive model which is trained with a
number of O(100K) music audio clips paired with video frames, which are mined
from in-the-wild music videos, and no parallel symbolic music data is involved.
V2Meow is able to synthesize high-fidelity music audio waveform solely
conditioned on pre-trained visual features extracted from an arbitrary silent
video clip, and it also allows high-level control over the music style of
generation examples via supporting text prompts in addition to the video frames
conditioning. Through both qualitative and quantitative evaluations, we
demonstrate that our model outperforms several existing music generation
systems in terms of both visual-audio correspondence and audio quality.
645d9bdff5760d1530d6b66a
Kun Su
645d9bdff5760d1530d6b66b
Judith Yue Li
admin_assigned
2023-05-24T08:57:54.072Z
5f431c9f79c1ba4c353d0d7f
/avatars/7ac0936f469e63cfe8ce15849e128a43.svg
Judith Yue Li
user
whereisdodo
645d9bdff5760d1530d6b66c
Qingqing Huang
645d9bdff5760d1530d6b66d
Dima Kuzmin
645d9bdff5760d1530d6b66e
Joonseok Lee
645d9bdff5760d1530d6b66f
Chris Donahue
admin_assigned
2023-05-24T08:58:41.505Z
63b1b5f034b91b8272aac6c8
/avatars/155351aa15967b5346144c03fc87d1ca.svg
Chris Donahue
user
chrisdonahue
645d9bdff5760d1530d6b670
Fei Sha
645d9bdff5760d1530d6b671
Aren Jansen
645d9bdff5760d1530d6b672
Yu Wang
645d9bdff5760d1530d6b673
Mauro Verzetti
645d9bdff5760d1530d6b674
Timo I. Denk
V2Meow: Meowing to the Visual Beat via Music Generation
Generating high quality music that complements the visual content of a video
is a challenging task. Most existing visual conditioned music generation
systems generate symbolic music data, such as MIDI files, instead of raw audio
waveform. Given the limited availability of symbolic music data, such methods
can only generate music for a few instruments or for specific types of visual
input. In this paper, we propose a novel approach called V2Meow that can
generate high-quality music audio that aligns well with the visual semantics of
a diverse range of video input types. Specifically, the proposed music
generation system is a multi-stage autoregressive model which is trained with a
number of O(100K) music audio clips paired with video frames, which are mined
from in-the-wild music videos, and no parallel symbolic music data is involved.
V2Meow is able to synthesize high-fidelity music audio waveform solely
conditioned on pre-trained visual features extracted from an arbitrary silent
video clip, and it also allows high-level control over the music style of
generation examples via supporting text prompts in addition to the video frames
conditioning. Through both qualitative and quantitative evaluations, we
demonstrate that our model outperforms several existing music generation
systems in terms of both visual-audio correspondence and audio quality.
A novel multi-stage autoregressive model, V2Meow, generates high-quality music audio aligned with various video inputs, outperforming existing methods in visual-audio correspondence and audio quality.
multi-stage autoregressive model high-fidelity music audio visual-audio correspondence audio quality pre-trained visual features text prompts