Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding
<|reference_start|>Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding: Large language models (LLMs) have demonstrated remarkable capabilities in natural language and multimodal domains. By fine-tuning multimodal LLMs with temporal annotations from well-annotated datasets, e.g., dense video captioning datasets, their temporal understanding capacity in video-language tasks can be obtained. However, there is a notable lack of untrimmed audio-visual video datasets with precise temporal annotations for events. This deficiency hinders LLMs from learning the alignment between time, audio-visual events, and text tokens, thus impairing their ability to temporally localize audio-visual events in videos. To address this gap, we introduce PU-VALOR, a comprehensive audio-visual dataset comprising over 114,000 pseudo-untrimmed videos with detailed temporal annotations. PU-VALOR is derived from the large-scale but coarse-annotated audio-visual dataset VALOR, through a subtle method involving event-based video clustering, random temporal scaling, and permutation. By fine-tuning a multimodal LLM on PU-VALOR, we developed AVicuna, a model capable of aligning audio-visual events with temporal intervals and corresponding text tokens. AVicuna excels in temporal localization and time-aware dialogue capabilities. Our experiments demonstrate that AVicuna effectively handles temporal understanding in audio-visual videos and achieves state-of-the-art performance on open-ended video QA, audio-visual QA, and audio-visual event dense localization tasks.<|reference_end|>
@article{tang2024empowering,
    title={Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal
  Understanding},
    author={Yunlong Tang, Daiki Shimada, Jing Bi, Mingqian Feng, Hang Hua,
  Chenliang Xu},
    journal={arXiv preprint arXiv:2403.16276},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.16276},
    primaryClass={cs.CV cs.AI}
}