SpeakerBeam-SS: Real-time Target Speaker Extraction with Lightweight Conv-TasNet and State Space Modeling
<|reference_start|>SpeakerBeam-SS: Real-time Target Speaker Extraction with Lightweight Conv-TasNet and State Space Modeling: Real-time target speaker extraction (TSE) is intended to extract the desired speaker's voice from the observed mixture of multiple speakers in a streaming manner. Implementing real-time TSE is challenging as the computational complexity must be reduced to provide real-time operation. This work introduces to Conv-TasNet-based TSE a new architecture based on state space modeling (SSM) that has been shown to model long-term dependency effectively. Owing to SSM, fewer dilated convolutional layers are required to capture temporal dependency in Conv-TasNet, resulting in the reduction of model complexity. We also enlarge the window length and shift of the convolutional (TasNet) frontend encoder to reduce the computational cost further; the performance decline is compensated by over-parameterization of the frontend encoder. The proposed method reduces the real-time factor by 78% from the conventional causal Conv-TasNet-based TSE while matching its performance.<|reference_end|>
@article{sato2024speakerbeam-ss:,
    title={SpeakerBeam-SS: Real-time Target Speaker Extraction with Lightweight
  Conv-TasNet and State Space Modeling},
    author={Hiroshi Sato, Takafumi Moriya, Masato Mimura, Shota Horiguchi, Tsubasa
  Ochiai, Takanori Ashihara, Atsushi Ando, Kentaro Shinayama, Marc Delcroix},
    journal={arXiv preprint arXiv:2407.01857},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.01857},
    primaryClass={eess.AS cs.SD eess.SP}
}