Learning Real-World Action-Video Dynamics with Heterogeneous Masked
  Autoregression
We propose Heterogeneous Masked Autoregression (HMA) for modeling
action-video dynamics to generate high-quality data and evaluation in scaling
robot learning. Building interactive video world models and policies for
robotics is difficult due to the challenge of handling diverse settings while
maintaining computational efficiency to run in real time. HMA uses
heterogeneous pre-training from observations and action sequences across
different robotic embodiments, domains, and tasks. HMA uses masked
autoregression to generate quantized or soft tokens for video predictions.
\ourshort achieves better visual fidelity and controllability than the previous
robotic video generation models with 15 times faster speed in the real world.
After post-training, this model can be used as a video simulator from low-level
action inputs for evaluating policies and generating synthetic data. See this
link https://liruiw.github.io/hma for more information.
67a57a4637e2abc28667ec1b
Lirui Wang
admin_assigned
2025-02-07T10:51:11.182Z
63151385b031f7b1c7c0871c
/avatars/0088eb929866face5f95218943e3f478.svg
Lirui Wang
user
liruiw
67a57a4637e2abc28667ec1c
Kevin Zhao
67a57a4637e2abc28667ec1d
Chaoqi Liu
admin_assigned
2025-02-07T10:51:27.451Z
6747a05a736eaadf2eec50ff
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/mF6_-m3GRm5OfG2HDNorC.jpeg
Chaoqi Liu
user
chaoqi-liu
67a57a4637e2abc28667ec1e
Xinlei Chen
admin_assigned
2025-02-07T10:51:33.905Z
63e58e3a006a775275e59e41
/avatars/75262a35b27a2ae1939df9118120d99e.svg
Xinlei Chen
user
endernewton
Learning Real-World Action-Video Dynamics with Heterogeneous Masked
  Autoregression
We propose Heterogeneous Masked Autoregression (HMA) for modeling
action-video dynamics to generate high-quality data and evaluation in scaling
robot learning. Building interactive video world models and policies for
robotics is difficult due to the challenge of handling diverse settings while
maintaining computational efficiency to run in real time. HMA uses
heterogeneous pre-training from observations and action sequences across
different robotic embodiments, domains, and tasks. HMA uses masked
autoregression to generate quantized or soft tokens for video predictions.
\ourshort achieves better visual fidelity and controllability than the previous
robotic video generation models with 15 times faster speed in the real world.
After post-training, this model can be used as a video simulator from low-level
action inputs for evaluating policies and generating synthetic data. See this
link https://liruiw.github.io/hma for more information.
Heterogeneous Masked Autoregression (HMA) improves robotic video generation by using heterogeneous pre-training and masked autoregression, delivering high visual fidelity, controllability, and speed.
Heterogeneous Masked Autoregression HMA masked autoregression quantized tokens soft tokens video predictions video simulator robotic video generation