Scene Adaptive Sparse Transformer for Event-based Object Detection
<|reference_start|>Scene Adaptive Sparse Transformer for Event-based Object Detection: While recent Transformer-based approaches have shown impressive performances on event-based object detection tasks, their high computational costs still diminish the low power consumption advantage of event cameras. Image-based works attempt to reduce these costs by introducing sparse Transformers. However, they display inadequate sparsity and adaptability when applied to event-based object detection, since these approaches cannot balance the fine granularity of token-level sparsification and the efficiency of window-based Transformers, leading to reduced performance and efficiency. Furthermore, they lack scene-specific sparsity optimization, resulting in information loss and a lower recall rate. To overcome these limitations, we propose the Scene Adaptive Sparse Transformer (SAST). SAST enables window-token co-sparsification, significantly enhancing fault tolerance and reducing computational overhead. Leveraging the innovative scoring and selection modules, along with the Masked Sparse Window Self-Attention, SAST showcases remarkable scene-aware adaptability: It focuses only on important objects and dynamically optimizes sparsity level according to scene complexity, maintaining a remarkable balance between performance and computational cost. The evaluation results show that SAST outperforms all other dense and sparse networks in both performance and efficiency on two large-scale event-based object detection datasets (1Mpx and Gen1). Code: https://github.com/Peterande/SAST<|reference_end|>
@article{peng2024scene,
    title={Scene Adaptive Sparse Transformer for Event-based Object Detection},
    author={Yansong Peng, Hebei Li, Yueyi Zhang, Xiaoyan Sun, Feng Wu},
    journal={arXiv preprint arXiv:2404.01882},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.01882},
    primaryClass={cs.CV}
}