title: Latent Action Pretraining from Videos
summary: We introduce Latent Action Pretraining for general Action models (LAPA), an
  unsupervised method for pretraining Vision-Language-Action (VLA) models without
  ground-truth robot action labels. Existing Vision-Language-Action models
  require action labels typically collected by human teleoperators during
  pretraining, which significantly limits possible data sources and scale. In
  this work, we propose a method to learn from internet-scale videos that do not
  have robot action labels. We first train an action quantization model
  leveraging VQ-VAE-based objective to learn discrete latent actions between
  image frames, then pretrain a latent VLA model to predict these latent actions
  from observations and task descriptions, and finally finetune the VLA on
  small-scale robot manipulation data to map from latent to robot actions.
  Experimental results demonstrate that our method significantly outperforms
  existing techniques that train robot manipulation policies from large-scale
  videos. Furthermore, it outperforms the state-of-the-art VLA model trained with
  robotic action labels on real-world manipulation tasks that require language
  conditioning, generalization to unseen objects, and semantic generalization to
  unseen instructions. Training only on human manipulation videos also shows
  positive transfer, opening up the potential for leveraging web-scale data for
  robotics foundation model.
paper_authors._id: 670f15d6cdc5f5a980175c6a
paper_authors.name: Seonghyeon Ye
paper_authors._id: 670f15d6cdc5f5a980175c6b
paper_authors.name: Joel Jang
paper_authors._id: 670f15d6cdc5f5a980175c6c
paper_authors.name: Byeongguk Jeon
paper_authors._id: 670f15d6cdc5f5a980175c6d
paper_authors.name: Sejune Joo
paper_authors._id: 670f15d6cdc5f5a980175c6e
paper_authors.name: Jianwei Yang
paper_authors._id: 670f15d6cdc5f5a980175c6f
paper_authors.name: Baolin Peng
paper_authors._id: 670f15d6cdc5f5a980175c70
paper_authors.name: Ajay Mandlekar
paper_authors._id: 670f15d6cdc5f5a980175c71
paper_authors.name: Reuben Tan
paper_authors._id: 670f15d6cdc5f5a980175c72
paper_authors.name: Yu-Wei Chao
paper_authors._id: 670f15d6cdc5f5a980175c73
paper_authors.name: Bill Yuchen Lin
paper_authors._id: 670f15d6cdc5f5a980175c74
paper_authors.name: Lars Liden
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T16:15:28.270Z
paper_authors._id: 65bc13c7719492167d777718
paper_authors.avatarUrl: /avatars/d1a3dc52c130b84a47c8a4ddd2e74be8.svg
paper_authors.fullname: Lars Liden
paper_authors.type: user
paper_authors.user: larsliden
paper_authors._id: 670f15d6cdc5f5a980175c75
paper_authors.name: Kimin Lee
paper_authors._id: 670f15d6cdc5f5a980175c76
paper_authors.name: Jianfeng Gao
paper_authors._id: 670f15d6cdc5f5a980175c77
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 670f15d6cdc5f5a980175c78
paper_authors.name: Dieter Fox
paper_authors._id: 670f15d6cdc5f5a980175c79
paper_authors.name: Minjoon Seo
paper_title: Latent Action Pretraining from Videos
paper_summary: We introduce Latent Action Pretraining for general Action models (LAPA), an
  unsupervised method for pretraining Vision-Language-Action (VLA) models without
  ground-truth robot action labels. Existing Vision-Language-Action models
  require action labels typically collected by human teleoperators during
  pretraining, which significantly limits possible data sources and scale. In
  this work, we propose a method to learn from internet-scale videos that do not
  have robot action labels. We first train an action quantization model
  leveraging VQ-VAE-based objective to learn discrete latent actions between
  image frames, then pretrain a latent VLA model to predict these latent actions
  from observations and task descriptions, and finally finetune the VLA on
  small-scale robot manipulation data to map from latent to robot actions.
  Experimental results demonstrate that our method significantly outperforms
  existing techniques that train robot manipulation policies from large-scale
  videos. Furthermore, it outperforms the state-of-the-art VLA model trained with
  robotic action labels on real-world manipulation tasks that require language
  conditioning, generalization to unseen objects, and semantic generalization to
  unseen instructions. Training only on human manipulation videos also shows
  positive transfer, opening up the potential for leveraging web-scale data for
  robotics foundation model.
paper_ai_keywords: latent Action Pretraining (LAPA) Vision-Language-Action (VLA) models unsupervised method pretraining action labels latent VLA model action quantization model VQ-VAE-based objective discrete latent actions real-quality techniques robot manipulation policies state-of-the-art VLA model language conditioning semantic generalization human manipulation videos web-scale data robotics foundation model