Unified Video Action Model
A unified video and action model holds significant promise for robotics,
where videos provide rich scene information for action prediction, and actions
provide dynamics information for video prediction. However, effectively
combining video generation and action prediction remains challenging, and
current video generation-based methods struggle to match the performance of
direct policy learning in action accuracy and inference speed. To bridge this
gap, we introduce the Unified Video Action model (UVA), which jointly optimizes
video and action predictions to achieve both high accuracy and efficient action
inference. The key lies in learning a joint video-action latent representation
and decoupling video-action decoding. The joint latent representation bridges
the visual and action domains, effectively modeling the relationship between
video and action sequences. Meanwhile, the decoupled decoding, powered by two
lightweight diffusion heads, enables high-speed action inference by bypassing
video generation during inference. Such a unified framework further enables
versatile functionality through masked input training. By selectively masking
actions or videos, a single model can tackle diverse tasks beyond policy
learning, such as forward and inverse dynamics modeling and video generation.
Via an extensive set of experiments, we demonstrate that UVA can serve as a
general-purpose solution for a wide range of robotics tasks, such as policy
learning, forward/inverse dynamics and video observation prediction, without
compromising performance compared to methods tailored for specific
applications. Results are best viewed on
https://unified-video-action-model.github.io/.
67c8bdbbbece4af63c15f388
Shuang Li
67c8bdbbbece4af63c15f389
Yihuai Gao
67c8bdbbbece4af63c15f38a
Dorsa Sadigh
67c8bdbbbece4af63c15f38b
Shuran Song
Unified Video Action Model
A unified video and action model holds significant promise for robotics,
where videos provide rich scene information for action prediction, and actions
provide dynamics information for video prediction. However, effectively
combining video generation and action prediction remains challenging, and
current video generation-based methods struggle to match the performance of
direct policy learning in action accuracy and inference speed. To bridge this
gap, we introduce the Unified Video Action model (UVA), which jointly optimizes
video and action predictions to achieve both high accuracy and efficient action
inference. The key lies in learning a joint video-action latent representation
and decoupling video-action decoding. The joint latent representation bridges
the visual and action domains, effectively modeling the relationship between
video and action sequences. Meanwhile, the decoupled decoding, powered by two
lightweight diffusion heads, enables high-speed action inference by bypassing
video generation during inference. Such a unified framework further enables
versatile functionality through masked input training. By selectively masking
actions or videos, a single model can tackle diverse tasks beyond policy
learning, such as forward and inverse dynamics modeling and video generation.
Via an extensive set of experiments, we demonstrate that UVA can serve as a
general-purpose solution for a wide range of robotics tasks, such as policy
learning, forward/inverse dynamics and video observation prediction, without
compromising performance compared to methods tailored for specific
applications. Results are best viewed on
https://unified-video-action-model.github.io/.
The Unified Video Action (UVA) model integrates video and action predictions through a joint latent representation and decoupled decoding for high accuracy and efficient action inference in robotics.
Unified Video Action model UVA joint latent representation decoupled decoding diffusion heads masked input training forward dynamics modeling inverse dynamics modeling video generation robotics tasks