title: BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural
    Speech Synthesis with Flow Matching Models
summary: Binaural rendering aims to synthesize binaural audio that mimics natural
  hearing based on a mono audio and the locations of the speaker and listener.
  Although many methods have been proposed to solve this problem, they struggle
  with rendering quality and streamable inference. Synthesizing high-quality
  binaural audio that is indistinguishable from real-world recordings requires
  precise modeling of binaural cues, room reverb, and ambient sounds.
  Additionally, real-world applications demand streaming inference. To address
  these challenges, we propose a flow matching based streaming binaural speech
  synthesis framework called BinauralFlow. We consider binaural rendering to be a
  generation problem rather than a regression problem and design a conditional
  flow matching model to render high-quality audio. Moreover, we design a causal
  U-Net architecture that estimates the current audio frame solely based on past
  information to tailor generative models for streaming inference. Finally, we
  introduce a continuous inference pipeline incorporating streaming STFT/ISTFT
  operations, a buffer bank, a midpoint solver, and an early skip schedule to
  improve rendering continuity and speed. Quantitative and qualitative
  evaluations demonstrate the superiority of our method over SOTA approaches. A
  perceptual study further reveals that our model is nearly indistinguishable
  from real-world recordings, with a 42% confusion rate.
paper_authors._id: 683f167c791f5015dfcbee13
paper_authors.name: Susan Liang
paper_authors._id: 683f167c791f5015dfcbee14
paper_authors.name: Dejan Markovic
paper_authors._id: 683f167c791f5015dfcbee15
paper_authors.name: Israel D. Gebru
paper_authors._id: 683f167c791f5015dfcbee16
paper_authors.name: Steven Krenn
paper_authors._id: 683f167c791f5015dfcbee17
paper_authors.name: Todd Keebler
paper_authors._id: 683f167c791f5015dfcbee18
paper_authors.name: Jacob Sandakly
paper_authors._id: 683f167c791f5015dfcbee19
paper_authors.name: Frank Yu
paper_authors._id: 683f167c791f5015dfcbee1a
paper_authors.name: Samuel Hassel
paper_authors._id: 683f167c791f5015dfcbee1b
paper_authors.name: Chenliang Xu
paper_authors._id: 683f167c791f5015dfcbee1c
paper_authors.name: Alexander Richard
paper_title: BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural
    Speech Synthesis with Flow Matching Models
paper_summary: Binaural rendering aims to synthesize binaural audio that mimics natural
  hearing based on a mono audio and the locations of the speaker and listener.
  Although many methods have been proposed to solve this problem, they struggle
  with rendering quality and streamable inference. Synthesizing high-quality
  binaural audio that is indistinguishable from real-world recordings requires
  precise modeling of binaural cues, room reverb, and ambient sounds.
  Additionally, real-world applications demand streaming inference. To address
  these challenges, we propose a flow matching based streaming binaural speech
  synthesis framework called BinauralFlow. We consider binaural rendering to be a
  generation problem rather than a regression problem and design a conditional
  flow matching model to render high-quality audio. Moreover, we design a causal
  U-Net architecture that estimates the current audio frame solely based on past
  information to tailor generative models for streaming inference. Finally, we
  introduce a continuous inference pipeline incorporating streaming STFT/ISTFT
  operations, a buffer bank, a midpoint solver, and an early skip schedule to
  improve rendering continuity and speed. Quantitative and qualitative
  evaluations demonstrate the superiority of our method over SOTA approaches. A
  perceptual study further reveals that our model is nearly indistinguishable
  from real-world recordings, with a 42% confusion rate.
paper_ai_summary: A flow matching based streaming binaural speech synthesis framework called BinauralFlow generates high-quality, indistinguishable binaural audio using a causal U-Net architecture and continuous inference pipeline.
paper_ai_keywords: flow matching streaming binaural speech synthesis BinauralFlow conditional flow matching model causal U-Net continuous inference pipeline streaming STFT/ISTFT buffer bank midpoint solver early skip schedule