Patch Spatio-Temporal Relation Prediction for Video Anomaly Detection
<|reference_start|>Patch Spatio-Temporal Relation Prediction for Video Anomaly Detection: Video Anomaly Detection (VAD), aiming to identify abnormalities within a specific context and timeframe, is crucial for intelligent Video Surveillance Systems. While recent deep learning-based VAD models have shown promising results by generating high-resolution frames, they often lack competence in preserving detailed spatial and temporal coherence in video frames. To tackle this issue, we propose a self-supervised learning approach for VAD through an inter-patch relationship prediction task. Specifically, we introduce a two-branch vision transformer network designed to capture deep visual features of video frames, addressing spatial and temporal dimensions responsible for modeling appearance and motion patterns, respectively. The inter-patch relationship in each dimension is decoupled into inter-patch similarity and the order information of each patch. To mitigate memory consumption, we convert the order information prediction task into a multi-label learning problem, and the inter-patch similarity prediction task into a distance matrix regression problem. Comprehensive experiments demonstrate the effectiveness of our method, surpassing pixel-generation-based methods by a significant margin across three public benchmarks. Additionally, our approach outperforms other self-supervised learning-based methods.<|reference_end|>
@article{shen2024patch,
    title={Patch Spatio-Temporal Relation Prediction for Video Anomaly Detection},
    author={Hao Shen, Lu Shi, Wanru Xu, Yigang Cen, Linna Zhang, Gaoyun An},
    journal={arXiv preprint arXiv:2403.19111},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.19111},
    primaryClass={cs.CV}
}