Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation for Complex Scenes
<|reference_start|>Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation for Complex Scenes: Rapid and reliable identification of dynamic scene parts, also known as motion segmentation, is a key challenge for mobile sensors. Contemporary RGB camera-based methods rely on modeling camera and scene properties however, are often under-constrained and fall short in unknown categories. Event cameras have the potential to overcome these limitations, but corresponding methods have only been demonstrated in smaller-scale indoor environments with simplified dynamic objects. This work presents an event-based method for class-agnostic motion segmentation that can successfully be deployed across complex large-scale outdoor environments too. To this end, we introduce a novel divide-and-conquer pipeline that combines: (a) ego-motion compensated events, computed via a scene understanding module that predicts monocular depth and camera pose as auxiliary tasks, and (b) optical flow from a dedicated optical flow module. These intermediate representations are then fed into a segmentation module that predicts motion segmentation masks. A novel transformer-based temporal attention module in the segmentation module builds correlations across adjacent 'frames' to get temporally consistent segmentation masks. Our method sets the new state-of-the-art on the classic EV-IMO benchmark (indoors), where we achieve improvements of 2.19 moving object IoU (2.22 mIoU) and 4.52 point IoU respectively, as well as on a newly-generated motion segmentation and tracking benchmark (outdoors) based on the DSEC event dataset, termed DSEC-MOTS, where we show improvement of 12.91 moving object IoU.<|reference_end|>
@article{georgoulis2024out,
    title={Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation
  for Complex Scenes},
    author={Stamatios Georgoulis, Weining Ren, Alfredo Bochicchio, Daniel Eckert,
  Yuanyou Li, and Abel Gawel},
    journal={arXiv preprint arXiv:2403.04562},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.04562},
    primaryClass={cs.CV}
}