SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound
Generating combined visual and auditory sensory experiences is critical for
the consumption of immersive content. Recent advances in neural generative
models have enabled the creation of high-resolution content across multiple
modalities such as images, text, speech, and videos. Despite these successes,
there remains a significant gap in the generation of high-quality spatial audio
that complements generated visual content. Furthermore, current audio
generation models excel in either generating natural audio or speech or music
but fall short in integrating spatial audio cues necessary for immersive
experiences. In this work, we introduce SEE-2-SOUND, a zero-shot approach that
decomposes the task into (1) identifying visual regions of interest; (2)
locating these elements in 3D space; (3) generating mono-audio for each; and
(4) integrating them into spatial audio. Using our framework, we demonstrate
compelling results for generating spatial audio for high-quality videos,
images, and dynamic images from the internet, as well as media generated by
learned approaches.
66691ba8c4ef4175fb26ee97
Rishit Dagli
admin_assigned
2024-06-12T08:41:00.397Z
60796959c59d9e1697fa2324
https://cdn-avatars.huggingface.co/v1/production/uploads/60796959c59d9e1697fa2324/wxnDm-p3YgB95NV2p4LGF.png
Rishit Dagli
user
rishitdagli
66691ba8c4ef4175fb26ee98
Shivesh Prakash
claimed_verified
2024-06-13T09:31:52.746Z
6669b4eb01c274049de3b473
https://cdn-avatars.huggingface.co/v1/production/uploads/6669b4eb01c274049de3b473/6bxPf4xKswAvk7f_ncRrW.jpeg
Shivesh Prakash
user
Shivesh777
66691ba8c4ef4175fb26ee99
Robert Wu
claimed_verified
2024-06-13T09:31:50.502Z
63a16bf05d09b819fee97d94
https://cdn-avatars.huggingface.co/v1/production/uploads/63a16bf05d09b819fee97d94/yTSPoJEASst9ztmcjLY6m.jpeg
Rupert Wu
user
rhubarbwu
66691ba8c4ef4175fb26ee9a
Houman Khosravani
admin_assigned
2024-06-12T08:41:26.500Z
63dde9a4422ca8d7f7e7d6f9
/avatars/e1fb69477fc5f1a5356b7cb66d65bcde.svg
Houman Khosravani
user
houmank
SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound
Generating combined visual and auditory sensory experiences is critical for
the consumption of immersive content. Recent advances in neural generative
models have enabled the creation of high-resolution content across multiple
modalities such as images, text, speech, and videos. Despite these successes,
there remains a significant gap in the generation of high-quality spatial audio
that complements generated visual content. Furthermore, current audio
generation models excel in either generating natural audio or speech or music
but fall short in integrating spatial audio cues necessary for immersive
experiences. In this work, we introduce SEE-2-SOUND, a zero-shot approach that
decomposes the task into (1) identifying visual regions of interest; (2)
locating these elements in 3D space; (3) generating mono-audio for each; and
(4) integrating them into spatial audio. Using our framework, we demonstrate
compelling results for generating spatial audio for high-quality videos,
images, and dynamic images from the internet, as well as media generated by
learned approaches.
SEE-2-SOUND generates spatial audio for visual content by decomposing the task into identifying visual regions, locating them in 3D space, generating mono-audio for each, and integrating it into spatial audio, addressing a gap in current audio generation models.
neural generative models high-resolution content images text speech videos spatial audio zero-shot approach visual regions of interest 3D space mono-audio learned approaches