Incremental FastPitch: Chunk-based High Quality Text to Speech
Parallel text-to-speech models have been widely applied for real-time speech
synthesis, and they offer more controllability and a much faster synthesis
process compared with conventional auto-regressive models. Although parallel
models have benefits in many aspects, they become naturally unfit for
incremental synthesis due to their fully parallel architecture such as
transformer. In this work, we propose Incremental FastPitch, a novel FastPitch
variant capable of incrementally producing high-quality Mel chunks by improving
the architecture with chunk-based FFT blocks, training with receptive-field
constrained chunk attention masks, and inference with fixed size past model
states. Experimental results show that our proposal can produce speech quality
comparable to the parallel FastPitch, with a significant lower latency that
allows even lower response time for real-time speech applications.
65962c9f25b1c7a91068f44b
Muyang Du
65962c9f25b1c7a91068f44c
Chuan Liu
65962c9f25b1c7a91068f44d
Junjie Lai
Incremental FastPitch: Chunk-based High Quality Text to Speech
Parallel text-to-speech models have been widely applied for real-time speech
synthesis, and they offer more controllability and a much faster synthesis
process compared with conventional auto-regressive models. Although parallel
models have benefits in many aspects, they become naturally unfit for
incremental synthesis due to their fully parallel architecture such as
transformer. In this work, we propose Incremental FastPitch, a novel FastPitch
variant capable of incrementally producing high-quality Mel chunks by improving
the architecture with chunk-based FFT blocks, training with receptive-field
constrained chunk attention masks, and inference with fixed size past model
states. Experimental results show that our proposal can produce speech quality
comparable to the parallel FastPitch, with a significant lower latency that
allows even lower response time for real-time speech applications.
Incremental FastPitch improves real-time speech synthesis by enabling incremental Mel chunk generation with reduced latency.
Parallel text-to-speech FastPitch chunk-based FFT blocks chunk attention masks real-time speech synthesis incremental synthesis latency