title: MaskGWM: A Generalizable Driving World Model with Video Mask
    Reconstruction
summary: World models that forecast environmental changes from actions are vital for
  autonomous driving models with strong generalization. The prevailing driving
  world model mainly build on video prediction model. Although these models can
  produce high-fidelity video sequences with advanced diffusion-based generator,
  they are constrained by their predictive duration and overall generalization
  capabilities. In this paper, we explore to solve this problem by combining
  generation loss with MAE-style feature-level context learning. In particular,
  we instantiate this target with three key design: (1) A more scalable Diffusion
  Transformer (DiT) structure trained with extra mask construction task. (2) we
  devise diffusion-related mask tokens to deal with the fuzzy relations between
  mask reconstruction and generative diffusion process. (3) we extend mask
  construction task to spatial-temporal domain by utilizing row-wise mask for
  shifted self-attention rather than masked self-attention in MAE. Then, we adopt
  a row-wise cross-view module to align with this mask design. Based on above
  improvement, we propose MaskGWM: a Generalizable driving World Model embodied
  with Video Mask reconstruction. Our model contains two variants: MaskGWM-long,
  focusing on long-horizon prediction, and MaskGWM-mview, dedicated to multi-view
  generation. Comprehensive experiments on standard benchmarks validate the
  effectiveness of the proposed method, which contain normal validation of
  Nuscene dataset, long-horizon rollout of OpenDV-2K dataset and zero-shot
  validation of Waymo dataset. Quantitative metrics on these datasets show our
  method notably improving state-of-the-art driving world model.
paper_authors._id: 67b705d2ebee4662205c47f7
paper_authors.name: Jingcheng Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:06:17.841Z
paper_authors._id: 65d444b1ea28ba508b87ab01
paper_authors.avatarUrl: /avatars/5836c0d64ba3936e064faa8ff4d44de0.svg
paper_authors.fullname: Jingcheng Ni
paper_authors.type: user
paper_authors.user: kiranjc
paper_authors._id: 67b705d2ebee4662205c47f8
paper_authors.name: Yuxin Guo
paper_authors._id: 67b705d2ebee4662205c47f9
paper_authors.name: Yichen Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:23:40.466Z
paper_authors._id: 6572dcc6bbd6664053b1fa6b
paper_authors.avatarUrl: /avatars/aba29efd00bc41f14ce422f7807cd2c3.svg
paper_authors.fullname: Liu Yichen
paper_authors.type: user
paper_authors.user: lyclyc52
paper_authors._id: 67b705d2ebee4662205c47fa
paper_authors.name: Rui Chen
paper_authors._id: 67b705d2ebee4662205c47fb
paper_authors.name: Lewei Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:06:46.453Z
paper_authors._id: 65ead3ea908526a39082e641
paper_authors.avatarUrl: /avatars/dcf870695fd56b06ca03d82f831e9019.svg
paper_authors.fullname: Lewei Lu
paper_authors.type: user
paper_authors.user: luotto
paper_authors._id: 67b705d2ebee4662205c47fc
paper_authors.name: Zehuan Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:59:38.956Z
paper_authors._id: 65717368be66cd9b65a8201c
paper_authors.avatarUrl: /avatars/fe945828eec9ded4cfa3b89d48a64d90.svg
paper_authors.fullname: Wu Zehuan
paper_authors.type: user
paper_authors.user: wzhgba
paper_title: MaskGWM: A Generalizable Driving World Model with Video Mask
    Reconstruction
paper_summary: World models that forecast environmental changes from actions are vital for
  autonomous driving models with strong generalization. The prevailing driving
  world model mainly build on video prediction model. Although these models can
  produce high-fidelity video sequences with advanced diffusion-based generator,
  they are constrained by their predictive duration and overall generalization
  capabilities. In this paper, we explore to solve this problem by combining
  generation loss with MAE-style feature-level context learning. In particular,
  we instantiate this target with three key design: (1) A more scalable Diffusion
  Transformer (DiT) structure trained with extra mask construction task. (2) we
  devise diffusion-related mask tokens to deal with the fuzzy relations between
  mask reconstruction and generative diffusion process. (3) we extend mask
  construction task to spatial-temporal domain by utilizing row-wise mask for
  shifted self-attention rather than masked self-attention in MAE. Then, we adopt
  a row-wise cross-view module to align with this mask design. Based on above
  improvement, we propose MaskGWM: a Generalizable driving World Model embodied
  with Video Mask reconstruction. Our model contains two variants: MaskGWM-long,
  focusing on long-horizon prediction, and MaskGWM-mview, dedicated to multi-view
  generation. Comprehensive experiments on standard benchmarks validate the
  effectiveness of the proposed method, which contain normal validation of
  Nuscene dataset, long-horizon rollout of OpenDV-2K dataset and zero-shot
  validation of Waymo dataset. Quantitative metrics on these datasets show our
  method notably improving state-of-the-art driving world model.
paper_ai_keywords: diffusion-based generator Diffusion Transformer (DiT) MAE-style feature-level context learning mask reconstruction generative diffusion process shifted self-attention cross-view module MaskGWM MaskGWM-long MaskGWM-mview Nuscene dataset OpenDV-2K dataset Waymo dataset driving world model