title: TesserAct: Learning 4D Embodied World Models
summary: This paper presents an effective approach for learning novel 4D embodied
  world models, which predict the dynamic evolution of 3D scenes over time in
  response to an embodied agent's actions, providing both spatial and temporal
  consistency. We propose to learn a 4D world model by training on RGB-DN (RGB,
  Depth, and Normal) videos. This not only surpasses traditional 2D models by
  incorporating detailed shape, configuration, and temporal changes into their
  predictions, but also allows us to effectively learn accurate inverse dynamic
  models for an embodied agent. Specifically, we first extend existing robotic
  manipulation video datasets with depth and normal information leveraging
  off-the-shelf models. Next, we fine-tune a video generation model on this
  annotated dataset, which jointly predicts RGB-DN (RGB, Depth, and Normal) for
  each frame. We then present an algorithm to directly convert generated RGB,
  Depth, and Normal videos into a high-quality 4D scene of the world. Our method
  ensures temporal and spatial coherence in 4D scene predictions from embodied
  scenarios, enables novel view synthesis for embodied environments, and
  facilitates policy learning that significantly outperforms those derived from
  prior video-based world models.
paper_authors._id: 68118c049c2765c9323de70b
paper_authors.name: Haoyu Zhen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T10:04:19.238Z
paper_authors._id: 6437c7dae282b4a48eaf065e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6437c7dae282b4a48eaf065e/AxodKQXyrviTFQRyjnL01.jpeg
paper_authors.fullname: Haoyu Zhen
paper_authors.type: user
paper_authors.user: anyeZHY
paper_authors._id: 68118c049c2765c9323de70c
paper_authors.name: Qiao Sun
paper_authors._id: 68118c049c2765c9323de70d
paper_authors.name: Hongxin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-02T06:36:47.235Z
paper_authors._id: 6319ef09bc8f3b313f7d04d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663253467220-6319ef09bc8f3b313f7d04d1.jpeg
paper_authors.fullname: Hongxin Zhang
paper_authors.type: user
paper_authors.user: Icefox
paper_authors._id: 68118c049c2765c9323de70e
paper_authors.name: Junyan Li
paper_authors._id: 68118c049c2765c9323de70f
paper_authors.name: Siyuan Zhou
paper_authors._id: 68118c049c2765c9323de710
paper_authors.name: Yilun Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T10:03:42.041Z
paper_authors._id: 63c9bd445fdc575773c732fe
paper_authors.avatarUrl: /avatars/def472d1ab3fbf751225357c0932ae7e.svg
paper_authors.fullname: Yilun Du
paper_authors.type: user
paper_authors.user: yilundu
paper_authors._id: 68118c049c2765c9323de711
paper_authors.name: Chuang Gan
paper_title: TesserAct: Learning 4D Embodied World Models
paper_summary: This paper presents an effective approach for learning novel 4D embodied
  world models, which predict the dynamic evolution of 3D scenes over time in
  response to an embodied agent's actions, providing both spatial and temporal
  consistency. We propose to learn a 4D world model by training on RGB-DN (RGB,
  Depth, and Normal) videos. This not only surpasses traditional 2D models by
  incorporating detailed shape, configuration, and temporal changes into their
  predictions, but also allows us to effectively learn accurate inverse dynamic
  models for an embodied agent. Specifically, we first extend existing robotic
  manipulation video datasets with depth and normal information leveraging
  off-the-shelf models. Next, we fine-tune a video generation model on this
  annotated dataset, which jointly predicts RGB-DN (RGB, Depth, and Normal) for
  each frame. We then present an algorithm to directly convert generated RGB,
  Depth, and Normal videos into a high-quality 4D scene of the world. Our method
  ensures temporal and spatial coherence in 4D scene predictions from embodied
  scenarios, enables novel view synthesis for embodied environments, and
  facilitates policy learning that significantly outperforms those derived from
  prior video-based world models.
paper_ai_summary: The paper introduces a 4D world model trained on RGB-DN videos to predict spatial and temporal dynamics in embodied environments, improving inverse dynamics learning, view synthesis, and policy performance.
paper_ai_keywords: 4D world model RGB-DN videos video generation model high-quality 4D scene temporal coherence spatial coherence embodied scenarios novel view synthesis policy learning