Rethinking RGB-Event Semantic Segmentation with a Novel Bidirectional
  Motion-enhanced Event Representation
Event cameras capture motion dynamics, offering a unique modality with great
potential in various computer vision tasks. However, RGB-Event fusion faces
three intrinsic misalignments: (i) temporal, (ii) spatial, and (iii) modal
misalignment. Existing voxel grid representations neglect temporal correlations
between consecutive event windows, and their formulation with simple
accumulation of asynchronous and sparse events is incompatible with the
synchronous and dense nature of RGB modality. To tackle these challenges, we
propose a novel event representation, Motion-enhanced Event Tensor (MET), which
transforms sparse event voxels into a dense and temporally coherent form by
leveraging dense optical flows and event temporal features. In addition, we
introduce a Frequency-aware Bidirectional Flow Aggregation Module (BFAM) and a
Temporal Fusion Module (TFM). BFAM leverages the frequency domain and MET to
mitigate modal misalignment, while bidirectional flow aggregation and temporal
fusion mechanisms resolve spatiotemporal misalignment. Experimental results on
two large-scale datasets demonstrate that our framework significantly
outperforms state-of-the-art RGB-Event semantic segmentation approaches. Our
code is available at: https://github.com/zyaocoder/BRENet.
681a662d26c8bff22d483fac
Zhen Yao
extracted_confirmed
2025-05-06T19:44:04.570Z
654939b935fa2ade2defa15b
https://cdn-avatars.huggingface.co/v1/production/uploads/654939b935fa2ade2defa15b/r-Ku79eVKxeCa1f8Ogf9C.png
Zhen Yao
user
Chrisathy
681a662d26c8bff22d483fad
Xiaowen Ying
681a662d26c8bff22d483fae
Mooi Choo Chuah
Rethinking RGB-Event Semantic Segmentation with a Novel Bidirectional
  Motion-enhanced Event Representation
Event cameras capture motion dynamics, offering a unique modality with great
potential in various computer vision tasks. However, RGB-Event fusion faces
three intrinsic misalignments: (i) temporal, (ii) spatial, and (iii) modal
misalignment. Existing voxel grid representations neglect temporal correlations
between consecutive event windows, and their formulation with simple
accumulation of asynchronous and sparse events is incompatible with the
synchronous and dense nature of RGB modality. To tackle these challenges, we
propose a novel event representation, Motion-enhanced Event Tensor (MET), which
transforms sparse event voxels into a dense and temporally coherent form by
leveraging dense optical flows and event temporal features. In addition, we
introduce a Frequency-aware Bidirectional Flow Aggregation Module (BFAM) and a
Temporal Fusion Module (TFM). BFAM leverages the frequency domain and MET to
mitigate modal misalignment, while bidirectional flow aggregation and temporal
fusion mechanisms resolve spatiotemporal misalignment. Experimental results on
two large-scale datasets demonstrate that our framework significantly
outperforms state-of-the-art RGB-Event semantic segmentation approaches. Our
code is available at: https://github.com/zyaocoder/BRENet.
A novel event representation and fusion framework enhances RGB-Event semantic segmentation by addressing temporal, spatial, and modal misalignments using dense optical flows and bidirectional flow aggregation.
voxel grid representations Motion-enhanced Event Tensor (MET) Frequency-aware Bidirectional Flow Aggregation Module (BFAM) Temporal Fusion Module (TFM) semantic segmentation dense optical flows bidirectional flow aggregation temporal fusion mechanisms