Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization
<|reference_start|>Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization: Audio-Visual Source Localization (AVSL) aims to locate sounding objects within video frames given the paired audio clips. Existing methods predominantly rely on self-supervised contrastive learning of audio-visual correspondence. Without any bounding-box annotations, they struggle to achieve precise localization, especially for small objects, and suffer from blurry boundaries and false positives. Moreover, the naive semi-supervised method is poor in fully leveraging the information of abundant unlabeled data. In this paper, we propose a novel semi-supervised learning framework for AVSL, namely Dual Mean-Teacher (DMT), comprising two teacher-student structures to circumvent the confirmation bias issue. Specifically, two teachers, pre-trained on limited labeled data, are employed to filter out noisy samples via the consensus between their predictions, and then generate high-quality pseudo-labels by intersecting their confidence maps. The sufficient utilization of both labeled and unlabeled data and the proposed unbiased framework enable DMT to outperform current state-of-the-art methods by a large margin, with CIoU of 90.4% and 48.8% on Flickr-SoundNet and VGG-Sound Source, obtaining 8.9%, 9.6% and 4.6%, 6.4% improvements over self- and semi-supervised methods respectively, given only 3% positional-annotations. We also extend our framework to some existing AVSL methods and consistently boost their performance.<|reference_end|>
@article{guo2024dual,
    title={Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for
  Audio-Visual Source Localization},
    author={Yuxin Guo, Shijie Ma, Hu Su, Zhiqing Wang, Yuhao Zhao, Wei Zou, Siyang
  Sun, Yun Zheng},
    journal={arXiv preprint arXiv:2403.03145},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.03145},
    primaryClass={cs.CV cs.LG cs.MM cs.SD eess.AS}
}