Long-term Pre-training for Temporal Action Detection with Transformers
<|reference_start|>Long-term Pre-training for Temporal Action Detection with Transformers: Temporal action detection (TAD) is challenging, yet fundamental for real-world video applications. Recently, DETR-based models for TAD have been prevailing thanks to their unique benefits. However, transformers demand a huge dataset, and unfortunately data scarcity in TAD causes a severe degeneration. In this paper, we identify two crucial problems from data scarcity: attention collapse and imbalanced performance. To this end, we propose a new pre-training strategy, Long-Term Pre-training (LTP), tailored for transformers. LTP has two main components: 1) class-wise synthesis, 2) long-term pretext tasks. Firstly, we synthesize long-form video features by merging video snippets of a target class and non-target classes. They are analogous to untrimmed data used in TAD, despite being created from trimmed data. In addition, we devise two types of long-term pretext tasks to learn long-term dependency. They impose long-term conditions such as finding second-to-fourth or short-duration actions. Our extensive experiments show state-of-the-art performances in DETR-based methods on ActivityNet-v1.3 and THUMOS14 by a large margin. Moreover, we demonstrate that LTP significantly relieves the data scarcity issues in TAD.<|reference_end|>
@article{kim2024long-term,
    title={Long-term Pre-training for Temporal Action Detection with Transformers},
    author={Jihwan Kim, Miso Lee, Jae-Pil Heo},
    journal={arXiv preprint arXiv:2408.13152},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.13152},
    primaryClass={cs.CV}
}