$R^2$-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding
<|reference_start|>$R^2$-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding: Video temporal grounding (VTG) is a fine-grained video understanding problem that aims to ground relevant clips in untrimmed videos given natural language queries. Most existing VTG models are built upon frame-wise final-layer CLIP features, aided by additional temporal backbones (e.g., SlowFast) with sophisticated temporal reasoning mechanisms. In this work, we claim that CLIP itself already shows great potential for fine-grained spatial-temporal modeling, as each layer offers distinct yet useful information under different granularity levels. Motivated by this, we propose Reversed Recurrent Tuning ($R^2$-Tuning), a parameter- and memory-efficient transfer learning framework for video temporal grounding. Our method learns a lightweight $R^2$ Block containing only 1.5% of the total parameters to perform progressive spatial-temporal modeling. Starting from the last layer of CLIP, $R^2$ Block recurrently aggregates spatial features from earlier layers, then refines temporal correlation conditioning on the given query, resulting in a coarse-to-fine scheme. $R^2$-Tuning achieves state-of-the-art performance across three VTG tasks (i.e., moment retrieval, highlight detection, and video summarization) on six public benchmarks (i.e., QVHighlights, Charades-STA, Ego4D-NLQ, TACoS, YouTube Highlights, and TVSum) even without the additional backbone, demonstrating the significance and effectiveness of the proposed scheme. Our code is available at https://github.com/yeliudev/R2-Tuning.<|reference_end|>
@article{liu2024$r^2$-tuning:,
    title={$R^2$-Tuning: Efficient Image-to-Video Transfer Learning for Video
  Temporal Grounding},
    author={Ye Liu, Jixuan He, Wanhua Li, Junsik Kim, Donglai Wei, Hanspeter
  Pfister, Chang Wen Chen},
    journal={arXiv preprint arXiv:2404.00801},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.00801},
    primaryClass={cs.CV}
}