title: Unified Video Action Model
summary: A unified video and action model holds significant promise for robotics,
  where videos provide rich scene information for action prediction, and actions
  provide dynamics information for video prediction. However, effectively
  combining video generation and action prediction remains challenging, and
  current video generation-based methods struggle to match the performance of
  direct policy learning in action accuracy and inference speed. To bridge this
  gap, we introduce the Unified Video Action model (UVA), which jointly optimizes
  video and action predictions to achieve both high accuracy and efficient action
  inference. The key lies in learning a joint video-action latent representation
  and decoupling video-action decoding. The joint latent representation bridges
  the visual and action domains, effectively modeling the relationship between
  video and action sequences. Meanwhile, the decoupled decoding, powered by two
  lightweight diffusion heads, enables high-speed action inference by bypassing
  video generation during inference. Such a unified framework further enables
  versatile functionality through masked input training. By selectively masking
  actions or videos, a single model can tackle diverse tasks beyond policy
  learning, such as forward and inverse dynamics modeling and video generation.
  Via an extensive set of experiments, we demonstrate that UVA can serve as a
  general-purpose solution for a wide range of robotics tasks, such as policy
  learning, forward/inverse dynamics and video observation prediction, without
  compromising performance compared to methods tailored for specific
  applications. Results are best viewed on
  https://unified-video-action-model.github.io/.
paper_authors._id: 67c8bdbbbece4af63c15f388
paper_authors.name: Shuang Li
paper_authors._id: 67c8bdbbbece4af63c15f389
paper_authors.name: Yihuai Gao
paper_authors._id: 67c8bdbbbece4af63c15f38a
paper_authors.name: Dorsa Sadigh
paper_authors._id: 67c8bdbbbece4af63c15f38b
paper_authors.name: Shuran Song
paper_title: Unified Video Action Model
paper_summary: A unified video and action model holds significant promise for robotics,
  where videos provide rich scene information for action prediction, and actions
  provide dynamics information for video prediction. However, effectively
  combining video generation and action prediction remains challenging, and
  current video generation-based methods struggle to match the performance of
  direct policy learning in action accuracy and inference speed. To bridge this
  gap, we introduce the Unified Video Action model (UVA), which jointly optimizes
  video and action predictions to achieve both high accuracy and efficient action
  inference. The key lies in learning a joint video-action latent representation
  and decoupling video-action decoding. The joint latent representation bridges
  the visual and action domains, effectively modeling the relationship between
  video and action sequences. Meanwhile, the decoupled decoding, powered by two
  lightweight diffusion heads, enables high-speed action inference by bypassing
  video generation during inference. Such a unified framework further enables
  versatile functionality through masked input training. By selectively masking
  actions or videos, a single model can tackle diverse tasks beyond policy
  learning, such as forward and inverse dynamics modeling and video generation.
  Via an extensive set of experiments, we demonstrate that UVA can serve as a
  general-purpose solution for a wide range of robotics tasks, such as policy
  learning, forward/inverse dynamics and video observation prediction, without
  compromising performance compared to methods tailored for specific
  applications. Results are best viewed on
  https://unified-video-action-model.github.io/.
paper_ai_summary: The Unified Video Action (UVA) model integrates video and action predictions through a joint latent representation and decoupled decoding for high accuracy and efficient action inference in robotics.
paper_ai_keywords: Unified Video Action model UVA joint latent representation decoupled decoding diffusion heads masked input training forward dynamics modeling inverse dynamics modeling video generation robotics tasks