title: StreamVoice: Streamable Context-Aware Language Modeling for Real-time
    Zero-Shot Voice Conversion
summary: Recent language model (LM) advancements have showcased impressive zero-shot
  voice conversion (VC) performance. However, existing LM-based VC models usually
  apply offline conversion from source semantics to acoustic features, demanding
  the complete source speech, and limiting their deployment to real-time
  applications. In this paper, we introduce StreamVoice, a novel streaming
  LM-based model for zero-shot VC, facilitating real-time conversion given
  arbitrary speaker prompts and source speech. Specifically, to enable streaming
  capability, StreamVoice employs a fully causal context-aware LM with a
  temporal-independent acoustic predictor, while alternately processing semantic
  and acoustic features at each time step of autoregression which eliminates the
  dependence on complete source speech. To address the potential performance
  degradation from the incomplete context in streaming processing, we enhance the
  context-awareness of the LM through two strategies: 1) teacher-guided context
  foresight, using a teacher model to summarize the present and future semantic
  context during training to guide the model's forecasting for missing context;
  2) semantic masking strategy, promoting acoustic prediction from preceding
  corrupted semantic and acoustic input, enhancing context-learning ability.
  Notably, StreamVoice is the first LM-based streaming zero-shot VC model without
  any future look-ahead. Experimental results demonstrate StreamVoice's streaming
  conversion capability while maintaining zero-shot performance comparable to
  non-streaming VC systems.
paper_authors._id: 65af44c87af20f8e929dc6c2
paper_authors.name: Zhichao Wang
paper_authors._id: 65af44c87af20f8e929dc6c3
paper_authors.name: Yuanzhe Chen
paper_authors._id: 65af44c87af20f8e929dc6c4
paper_authors.name: Xinsheng Wang
paper_authors._id: 65af44c87af20f8e929dc6c5
paper_authors.name: Zhuo Chen
paper_authors._id: 65af44c87af20f8e929dc6c6
paper_authors.name: Lei Xie
paper_authors._id: 65af44c87af20f8e929dc6c7
paper_authors.name: Yuping Wang
paper_authors._id: 65af44c87af20f8e929dc6c8
paper_authors.name: Yuxuan Wang
paper_title: StreamVoice: Streamable Context-Aware Language Modeling for Real-time
    Zero-Shot Voice Conversion
paper_summary: Recent language model (LM) advancements have showcased impressive zero-shot
  voice conversion (VC) performance. However, existing LM-based VC models usually
  apply offline conversion from source semantics to acoustic features, demanding
  the complete source speech, and limiting their deployment to real-time
  applications. In this paper, we introduce StreamVoice, a novel streaming
  LM-based model for zero-shot VC, facilitating real-time conversion given
  arbitrary speaker prompts and source speech. Specifically, to enable streaming
  capability, StreamVoice employs a fully causal context-aware LM with a
  temporal-independent acoustic predictor, while alternately processing semantic
  and acoustic features at each time step of autoregression which eliminates the
  dependence on complete source speech. To address the potential performance
  degradation from the incomplete context in streaming processing, we enhance the
  context-awareness of the LM through two strategies: 1) teacher-guided context
  foresight, using a teacher model to summarize the present and future semantic
  context during training to guide the model's forecasting for missing context;
  2) semantic masking strategy, promoting acoustic prediction from preceding
  corrupted semantic and acoustic input, enhancing context-learning ability.
  Notably, StreamVoice is the first LM-based streaming zero-shot VC model without
  any future look-ahead. Experimental results demonstrate StreamVoice's streaming
  conversion capability while maintaining zero-shot performance comparable to
  non-streaming VC systems.
paper_ai_summary: StreamVoice is a real-time, zero-shot voice conversion model that uses a fully causal context-aware language model with strategies to handle missing context, enabling conversion without future look-ahead.
paper_ai_keywords: zero-shot voice conversion streaming fully causal context-aware temporal-independent acoustic predictor autoregression teacher-guided context foresight semantic masking strategy