UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization
<|reference_start|>UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization: Video localization tasks aim to temporally locate specific instances in videos, including temporal action localization (TAL), sound event detection (SED) and audio-visual event localization (AVEL). Existing methods over-specialize on each task, overlooking the fact that these instances often occur in the same video to form the complete video content. In this work, we present UniAV, a Unified Audio-Visual perception network, to achieve joint learning of TAL, SED and AVEL tasks for the first time. UniAV can leverage diverse data available in task-specific datasets, allowing the model to learn and share mutually beneficial knowledge across tasks and modalities. To tackle the challenges posed by substantial variations in datasets (size/domain/duration) and distinct task characteristics, we propose to uniformly encode visual and audio modalities of all videos to derive generic representations, while also designing task-specific experts to capture unique knowledge for each task. Besides, we develop a unified language-aware classifier by utilizing a pre-trained text encoder, enabling the model to flexibly detect various types of instances and previously unseen ones by simply changing prompts during inference. UniAV outperforms its single-task counterparts by a large margin with fewer parameters, achieving on-par or superior performances compared to state-of-the-art task-specific methods across ActivityNet 1.3, DESED and UnAV-100 benchmarks.<|reference_end|>
@article{geng2024uniav:,
    title={UniAV: Unified Audio-Visual Perception for Multi-Task Video Event
  Localization},
    author={Tiantian Geng, Teng Wang, Yanfu Zhang, Jinming Duan, Weili Guan, Feng
  Zheng, Ling shao},
    journal={arXiv preprint arXiv:2404.03179},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.03179},
    primaryClass={cs.CV cs.MM cs.SD eess.AS}
}