Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction
M. Eren Akbiyik Nedko Savov Danda Pani Paudel Nikola Popovic Christian Vater Otmar Hilliges Luc Van Gool Xi Wang
Understanding drivers’ decision-making is crucial for road safety. Although predicting the ego-vehicle’s path is valuable for driver-assistance systems, existing methods mainly focus on external factors like other vehicles’ motions, often neglecting the driver’s attention and intent. To address this gap, we infer the ego-trajectory by integrating the driver’s gaze and the surrounding scene. We introduce RouteFormer, a novel multimodal ego-trajectory prediction network combining GPS data, environmental context, and the driver's field-of-view—comprising first-person video and gaze fixations. We also present the Path Complexity Index (PCI), a new metric for trajectory complexity that enables a more nuanced evaluation of challenging scenarios. To tackle data scarcity and enhance diversity, we introduce GEM, a comprehensive dataset of urban driving scenarios enriched with synchronized driver field-of-view and gaze data. Extensive evaluations on GEM and DR(eye)VE demonstrate that RouteFormer significantly outperforms state-of-the-art methods, achieving notable improvements in prediction accuracy across diverse conditions. Ablation studies reveal that incorporating driver field-of-view data yields significantly better average displacement error, especially in challenging scenarios with high PCI scores, underscoring the importance of modeling driver attention. All data and code are available at meakbiyik.github.io/routeformer.
Ego-trajectory prediction, driver attention, multimodal learning, field-of-view, gaze fixations, deep learning, autonomous driving, driver behavior modeling, dataset creation
Introduces RouteFormer, a multimodal ego-trajectory prediction model leveraging driver attention and field-of-view data, along with the GEM dataset and Path Complexity Index, significantly improving prediction accuracy over state-of-the-art methods.