title: Learning Real-World Action-Video Dynamics with Heterogeneous Masked
    Autoregression
summary: We propose Heterogeneous Masked Autoregression (HMA) for modeling
  action-video dynamics to generate high-quality data and evaluation in scaling
  robot learning. Building interactive video world models and policies for
  robotics is difficult due to the challenge of handling diverse settings while
  maintaining computational efficiency to run in real time. HMA uses
  heterogeneous pre-training from observations and action sequences across
  different robotic embodiments, domains, and tasks. HMA uses masked
  autoregression to generate quantized or soft tokens for video predictions.
  \ourshort achieves better visual fidelity and controllability than the previous
  robotic video generation models with 15 times faster speed in the real world.
  After post-training, this model can be used as a video simulator from low-level
  action inputs for evaluating policies and generating synthetic data. See this
  link https://liruiw.github.io/hma for more information.
paper_authors._id: 67a57a4637e2abc28667ec1b
paper_authors.name: Lirui Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:51:11.182Z
paper_authors._id: 63151385b031f7b1c7c0871c
paper_authors.avatarUrl: /avatars/0088eb929866face5f95218943e3f478.svg
paper_authors.fullname: Lirui Wang
paper_authors.type: user
paper_authors.user: liruiw
paper_authors._id: 67a57a4637e2abc28667ec1c
paper_authors.name: Kevin Zhao
paper_authors._id: 67a57a4637e2abc28667ec1d
paper_authors.name: Chaoqi Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:51:27.451Z
paper_authors._id: 6747a05a736eaadf2eec50ff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/mF6_-m3GRm5OfG2HDNorC.jpeg
paper_authors.fullname: Chaoqi Liu
paper_authors.type: user
paper_authors.user: chaoqi-liu
paper_authors._id: 67a57a4637e2abc28667ec1e
paper_authors.name: Xinlei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:51:33.905Z
paper_authors._id: 63e58e3a006a775275e59e41
paper_authors.avatarUrl: /avatars/75262a35b27a2ae1939df9118120d99e.svg
paper_authors.fullname: Xinlei Chen
paper_authors.type: user
paper_authors.user: endernewton
paper_title: Learning Real-World Action-Video Dynamics with Heterogeneous Masked
    Autoregression
paper_summary: We propose Heterogeneous Masked Autoregression (HMA) for modeling
  action-video dynamics to generate high-quality data and evaluation in scaling
  robot learning. Building interactive video world models and policies for
  robotics is difficult due to the challenge of handling diverse settings while
  maintaining computational efficiency to run in real time. HMA uses
  heterogeneous pre-training from observations and action sequences across
  different robotic embodiments, domains, and tasks. HMA uses masked
  autoregression to generate quantized or soft tokens for video predictions.
  \ourshort achieves better visual fidelity and controllability than the previous
  robotic video generation models with 15 times faster speed in the real world.
  After post-training, this model can be used as a video simulator from low-level
  action inputs for evaluating policies and generating synthetic data. See this
  link https://liruiw.github.io/hma for more information.
paper_ai_keywords: Heterogeneous Masked Autoregression (HMA) action-video dynamics real-time pre-training heterogeneous masked autoregression quantized tokens soft tokens video predictions visual fidelity controllability video simulator synthetic data