title: Moto: Latent Motion Token as the Bridging Language for Robot
    Manipulation
summary: Recent developments in Large Language Models pre-trained on extensive corpora
  have shown significant success in various natural language processing tasks
  with minimal fine-tuning. This success offers new promise for robotics, which
  has long been constrained by the high cost of action-labeled data. We ask:
  given the abundant video data containing interaction-related knowledge
  available as a rich "corpus", can a similar generative pre-training approach be
  effectively applied to enhance robot learning? The key challenge is to identify
  an effective representation for autoregressive pre-training that benefits robot
  manipulation tasks. Inspired by the way humans learn new skills through
  observing dynamic environments, we propose that effective robotic learning
  should emphasize motion-related knowledge, which is closely tied to low-level
  actions and is hardware-agnostic, facilitating the transfer of learned motions
  to actual robot actions. To this end, we introduce Moto, which converts video
  content into latent Motion Token sequences by a Latent Motion Tokenizer,
  learning a bridging "language" of motion from videos in an unsupervised manner.
  We pre-train Moto-GPT through motion token autoregression, enabling it to
  capture diverse visual motion knowledge. After pre-training, Moto-GPT
  demonstrates the promising ability to produce semantically interpretable motion
  tokens, predict plausible motion trajectories, and assess trajectory
  rationality through output likelihood. To transfer learned motion priors to
  real robot actions, we implement a co-fine-tuning strategy that seamlessly
  bridges latent motion token prediction and real robot control. Extensive
  experiments show that the fine-tuned Moto-GPT exhibits superior robustness and
  efficiency on robot manipulation benchmarks, underscoring its effectiveness in
  transferring knowledge from video data to downstream visual manipulation tasks.
paper_authors._id: 6752718e86b182da998a033f
paper_authors.name: Yi Chen
paper_authors._id: 6752718e86b182da998a0340
paper_authors.name: Yuying Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T15:14:52.699Z
paper_authors._id: 6455cc8f654d8bccae50e4d4
paper_authors.avatarUrl: /avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
paper_authors.fullname: Yuying Ge
paper_authors.type: user
paper_authors.user: tttoaster
paper_authors._id: 6752718e86b182da998a0341
paper_authors.name: Yizhuo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T15:14:46.404Z
paper_authors._id: 630b094f8b327c7b8b94d24c
paper_authors.avatarUrl: /avatars/f2e1e9b1c9af9856298f792a7872e224.svg
paper_authors.fullname: Yizhuo Li
paper_authors.type: user
paper_authors.user: liyz
paper_authors._id: 6752718e86b182da998a0342
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T15:14:40.299Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 6752718e86b182da998a0343
paper_authors.name: Mingyu Ding
paper_authors._id: 6752718e86b182da998a0344
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T15:14:16.664Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_authors._id: 6752718e86b182da998a0345
paper_authors.name: Xihui Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-09T15:14:02.421Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_title: Moto: Latent Motion Token as the Bridging Language for Robot
    Manipulation
paper_summary: Recent developments in Large Language Models pre-trained on extensive corpora
  have shown significant success in various natural language processing tasks
  with minimal fine-tuning. This success offers new promise for robotics, which
  has long been constrained by the high cost of action-labeled data. We ask:
  given the abundant video data containing interaction-related knowledge
  available as a rich "corpus", can a similar generative pre-training approach be
  effectively applied to enhance robot learning? The key challenge is to identify
  an effective representation for autoregressive pre-training that benefits robot
  manipulation tasks. Inspired by the way humans learn new skills through
  observing dynamic environments, we propose that effective robotic learning
  should emphasize motion-related knowledge, which is closely tied to low-level
  actions and is hardware-agnostic, facilitating the transfer of learned motions
  to actual robot actions. To this end, we introduce Moto, which converts video
  content into latent Motion Token sequences by a Latent Motion Tokenizer,
  learning a bridging "language" of motion from videos in an unsupervised manner.
  We pre-train Moto-GPT through motion token autoregression, enabling it to
  capture diverse visual motion knowledge. After pre-training, Moto-GPT
  demonstrates the promising ability to produce semantically interpretable motion
  tokens, predict plausible motion trajectories, and assess trajectory
  rationality through output likelihood. To transfer learned motion priors to
  real robot actions, we implement a co-fine-tuning strategy that seamlessly
  bridges latent motion token prediction and real robot control. Extensive
  experiments show that the fine-tuned Moto-GPT exhibits superior robustness and
  efficiency on robot manipulation benchmarks, underscoring its effectiveness in
  transferring knowledge from video data to downstream visual manipulation tasks.
paper_ai_keywords: Latent Motion Tokenizer Moto Moto-GPT motion token autoregression semantically interpretable motion tokens motion trajectories trajectory rationality output likelihood co-fine-tuning strategy robot manipulation benchmarks visual manipulation tasks