STNet: Deep Audio-Visual Fusion Network for Robust Speaker Tracking
<|reference_start|>STNet: Deep Audio-Visual Fusion Network for Robust Speaker Tracking: Audio-visual speaker tracking aims to determine the location of human targets in a scene using signals captured by a multi-sensor platform, whose accuracy and robustness can be improved by multi-modal fusion methods. Recently, several fusion methods have been proposed to model the correlation in multiple modalities. However, for the speaker tracking problem, the cross-modal interaction between audio and visual signals hasn't been well exploited. To this end, we present a novel Speaker Tracking Network (STNet) with a deep audio-visual fusion model in this work. We design a visual-guided acoustic measurement method to fuse heterogeneous cues in a unified localization space, which employs visual observations via a camera model to construct the enhanced acoustic map. For feature fusion, a cross-modal attention module is adopted to jointly model multi-modal contexts and interactions. The correlated information between audio and visual features is further interacted in the fusion model. Moreover, the STNet-based tracker is applied to multi-speaker cases by a quality-aware module, which evaluates the reliability of multi-modal observations to achieve robust tracking in complex scenarios. Experiments on the AV16.3 and CAV3D datasets show that the proposed STNet-based tracker outperforms uni-modal methods and state-of-the-art audio-visual speaker trackers.<|reference_end|>
@article{li2024stnet:,
    title={STNet: Deep Audio-Visual Fusion Network for Robust Speaker Tracking},
    author={Yidi Li and Hong Liu and Bing Yang},
    journal={arXiv preprint arXiv:2410.05964},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.05964},
    primaryClass={cs.CV cs.AI}
}