BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural
  Speech Synthesis with Flow Matching Models
Binaural rendering aims to synthesize binaural audio that mimics natural
hearing based on a mono audio and the locations of the speaker and listener.
Although many methods have been proposed to solve this problem, they struggle
with rendering quality and streamable inference. Synthesizing high-quality
binaural audio that is indistinguishable from real-world recordings requires
precise modeling of binaural cues, room reverb, and ambient sounds.
Additionally, real-world applications demand streaming inference. To address
these challenges, we propose a flow matching based streaming binaural speech
synthesis framework called BinauralFlow. We consider binaural rendering to be a
generation problem rather than a regression problem and design a conditional
flow matching model to render high-quality audio. Moreover, we design a causal
U-Net architecture that estimates the current audio frame solely based on past
information to tailor generative models for streaming inference. Finally, we
introduce a continuous inference pipeline incorporating streaming STFT/ISTFT
operations, a buffer bank, a midpoint solver, and an early skip schedule to
improve rendering continuity and speed. Quantitative and qualitative
evaluations demonstrate the superiority of our method over SOTA approaches. A
perceptual study further reveals that our model is nearly indistinguishable
from real-world recordings, with a 42% confusion rate.
683f167c791f5015dfcbee13
Susan Liang
683f167c791f5015dfcbee14
Dejan Markovic
683f167c791f5015dfcbee15
Israel D. Gebru
683f167c791f5015dfcbee16
Steven Krenn
683f167c791f5015dfcbee17
Todd Keebler
683f167c791f5015dfcbee18
Jacob Sandakly
683f167c791f5015dfcbee19
Frank Yu
683f167c791f5015dfcbee1a
Samuel Hassel
683f167c791f5015dfcbee1b
Chenliang Xu
683f167c791f5015dfcbee1c
Alexander Richard
BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural
  Speech Synthesis with Flow Matching Models
Binaural rendering aims to synthesize binaural audio that mimics natural
hearing based on a mono audio and the locations of the speaker and listener.
Although many methods have been proposed to solve this problem, they struggle
with rendering quality and streamable inference. Synthesizing high-quality
binaural audio that is indistinguishable from real-world recordings requires
precise modeling of binaural cues, room reverb, and ambient sounds.
Additionally, real-world applications demand streaming inference. To address
these challenges, we propose a flow matching based streaming binaural speech
synthesis framework called BinauralFlow. We consider binaural rendering to be a
generation problem rather than a regression problem and design a conditional
flow matching model to render high-quality audio. Moreover, we design a causal
U-Net architecture that estimates the current audio frame solely based on past
information to tailor generative models for streaming inference. Finally, we
introduce a continuous inference pipeline incorporating streaming STFT/ISTFT
operations, a buffer bank, a midpoint solver, and an early skip schedule to
improve rendering continuity and speed. Quantitative and qualitative
evaluations demonstrate the superiority of our method over SOTA approaches. A
perceptual study further reveals that our model is nearly indistinguishable
from real-world recordings, with a 42% confusion rate.
A flow matching based streaming binaural speech synthesis framework called BinauralFlow generates high-quality, indistinguishable binaural audio using a causal U-Net architecture and continuous inference pipeline.
flow matching streaming binaural speech synthesis BinauralFlow conditional flow matching model causal U-Net continuous inference pipeline streaming STFT/ISTFT buffer bank midpoint solver early skip schedule