title: SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound
summary: Generating combined visual and auditory sensory experiences is critical for
  the consumption of immersive content. Recent advances in neural generative
  models have enabled the creation of high-resolution content across multiple
  modalities such as images, text, speech, and videos. Despite these successes,
  there remains a significant gap in the generation of high-quality spatial audio
  that complements generated visual content. Furthermore, current audio
  generation models excel in either generating natural audio or speech or music
  but fall short in integrating spatial audio cues necessary for immersive
  experiences. In this work, we introduce SEE-2-SOUND, a zero-shot approach that
  decomposes the task into (1) identifying visual regions of interest; (2)
  locating these elements in 3D space; (3) generating mono-audio for each; and
  (4) integrating them into spatial audio. Using our framework, we demonstrate
  compelling results for generating spatial audio for high-quality videos,
  images, and dynamic images from the internet, as well as media generated by
  learned approaches.
paper_authors._id: 66691ba8c4ef4175fb26ee97
paper_authors.name: Rishit Dagli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:41:00.397Z
paper_authors._id: 60796959c59d9e1697fa2324
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60796959c59d9e1697fa2324/wxnDm-p3YgB95NV2p4LGF.png
paper_authors.fullname: Rishit Dagli
paper_authors.type: user
paper_authors.user: rishitdagli
paper_authors._id: 66691ba8c4ef4175fb26ee98
paper_authors.name: Shivesh Prakash
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:31:52.746Z
paper_authors._id: 6669b4eb01c274049de3b473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6669b4eb01c274049de3b473/6bxPf4xKswAvk7f_ncRrW.jpeg
paper_authors.fullname: Shivesh Prakash
paper_authors.type: user
paper_authors.user: Shivesh777
paper_authors._id: 66691ba8c4ef4175fb26ee99
paper_authors.name: Robert Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:31:50.502Z
paper_authors._id: 63a16bf05d09b819fee97d94
paper_authors.avatarUrl: /avatars/d9b97f66680ddce4dbc728b2cf51a3d1.svg
paper_authors.fullname: Robert Wu
paper_authors.type: user
paper_authors.user: rhubarbwu
paper_authors._id: 66691ba8c4ef4175fb26ee9a
paper_authors.name: Houman Khosravani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:41:26.500Z
paper_authors._id: 63dde9a4422ca8d7f7e7d6f9
paper_authors.avatarUrl: /avatars/e1fb69477fc5f1a5356b7cb66d65bcde.svg
paper_authors.fullname: Houman Khosravani
paper_authors.type: user
paper_authors.user: houmank
paper_title: SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound
paper_summary: Generating combined visual and auditory sensory experiences is critical for
  the consumption of immersive content. Recent advances in neural generative
  models have enabled the creation of high-resolution content across multiple
  modalities such as images, text, speech, and videos. Despite these successes,
  there remains a significant gap in the generation of high-quality spatial audio
  that complements generated visual content. Furthermore, current audio
  generation models excel in either generating natural audio or speech or music
  but fall short in integrating spatial audio cues necessary for immersive
  experiences. In this work, we introduce SEE-2-SOUND, a zero-shot approach that
  decomposes the task into (1) identifying visual regions of interest; (2)
  locating these elements in 3D space; (3) generating mono-audio for each; and
  (4) integrating them into spatial audio. Using our framework, we demonstrate
  compelling results for generating spatial audio for high-quality videos,
  images, and dynamic images from the internet, as well as media generated by
  learned approaches.
paper_ai_keywords: neural generative models high-resolution content modalities images text speech videos high-quality spatial audio immersive experiences zero-shot approach visual regions of interest 3D space mono-audio spatial audio