  Recently, the RGB images and point clouds fusion methods have been proposed
to jointly estimate 2D optical flow and 3D scene flow. However, as both
conventional RGB cameras and LiDAR sensors adopt a frame-based data acquisition
mechanism, their performance is limited by the fixed low sampling rates,
especially in highly-dynamic scenes. By contrast, the event camera can
asynchronously capture the intensity changes with a very high temporal
resolution, providing complementary dynamic information of the observed scenes.
In this paper, we incorporate RGB images, Point clouds and Events for joint
optical flow and scene flow estimation with our proposed multi-stage multimodal
fusion model, RPEFlow. First, we present an attention fusion module with a
cross-attention mechanism to implicitly explore the internal cross-modal
correlation for 2D and 3D branches, respectively. Second, we introduce a mutual
information regularization term to explicitly model the complementary
information of three modalities for effective multimodal feature learning. We
also contribute a new synthetic dataset to advocate further research.
Experiments on both synthetic and real datasets show that our model outperforms
the existing state-of-the-art by a wide margin. Code and dataset is available
at https://npucvr.github.io/RPEFlow.

Computer Vision and Pattern Recognition