title: Humanoid Locomotion as Next Token Prediction
summary: We cast real-world humanoid control as a next token prediction problem, akin
  to predicting the next word in language. Our model is a causal transformer
  trained via autoregressive prediction of sensorimotor trajectories. To account
  for the multi-modal nature of the data, we perform prediction in a
  modality-aligned way, and for each input token predict the next token from the
  same modality. This general formulation enables us to leverage data with
  missing modalities, like video trajectories without actions. We train our model
  on a collection of simulated trajectories coming from prior neural network
  policies, model-based controllers, motion capture data, and YouTube videos of
  humans. We show that our model enables a full-sized humanoid to walk in San
  Francisco zero-shot. Our model can transfer to the real world even when trained
  on only 27 hours of walking data, and can generalize to commands not seen
  during training like walking backward. These findings suggest a promising path
  toward learning challenging real-world control tasks by generative modeling of
  sensorimotor trajectories.
paper_authors._id: 65e16d5403a248cceb66bfd0
paper_authors.name: Ilija Radosavovic
paper_authors._id: 65e16d5403a248cceb66bfd1
paper_authors.name: Bike Zhang
paper_authors._id: 65e16d5403a248cceb66bfd2
paper_authors.name: Baifeng Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T09:25:37.870Z
paper_authors._id: 649004218f7cbbc94c782db6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
paper_authors.fullname: Baifeng Shi
paper_authors.type: user
paper_authors.user: bfshi
paper_authors._id: 65e16d5403a248cceb66bfd3
paper_authors.name: Jathushan Rajasegaran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:37:30.024Z
paper_authors._id: 63895b3a43d8b0797a0d0406
paper_authors.avatarUrl: /avatars/04d9952dff70f85d4e481ec80ae818cb.svg
paper_authors.fullname: Jathushan Rajasegaran
paper_authors.type: user
paper_authors.user: brjathu
paper_authors._id: 65e16d5403a248cceb66bfd4
paper_authors.name: Sarthak Kamat
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:41:28.529Z
paper_authors._id: 676edf92c1a36ee523b219c5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_nxPANpjOkUXQxrQhV-q7.png
paper_authors.fullname: Sarthak Kamat
paper_authors.type: user
paper_authors.user: sarthakkamat
paper_authors._id: 65e16d5403a248cceb66bfd5
paper_authors.name: Trevor Darrell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:37:41.172Z
paper_authors._id: 64cbdf02f103036e23d1c7f3
paper_authors.avatarUrl: /avatars/496069463900dea20929b57381182d39.svg
paper_authors.fullname: Trevor Darrell
paper_authors.type: user
paper_authors.user: trevordarrell
paper_authors._id: 65e16d5403a248cceb66bfd6
paper_authors.name: Koushil Sreenath
paper_authors._id: 65e16d5403a248cceb66bfd7
paper_authors.name: Jitendra Malik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:38:10.533Z
paper_authors._id: 65369a95605a07338de78ab0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
paper_authors.fullname: Jitendra Malik 
paper_authors.type: user
paper_authors.user: jitendra1995
paper_title: Humanoid Locomotion as Next Token Prediction
paper_summary: We cast real-world humanoid control as a next token prediction problem, akin
  to predicting the next word in language. Our model is a causal transformer
  trained via autoregressive prediction of sensorimotor trajectories. To account
  for the multi-modal nature of the data, we perform prediction in a
  modality-aligned way, and for each input token predict the next token from the
  same modality. This general formulation enables us to leverage data with
  missing modalities, like video trajectories without actions. We train our model
  on a collection of simulated trajectories coming from prior neural network
  policies, model-based controllers, motion capture data, and YouTube videos of
  humans. We show that our model enables a full-sized humanoid to walk in San
  Francisco zero-shot. Our model can transfer to the real world even when trained
  on only 27 hours of walking data, and can generalize to commands not seen
  during training like walking backward. These findings suggest a promising path
  toward learning challenging real-world control tasks by generative modeling of
  sensorimotor trajectories.
paper_ai_summary: A causal transformer for next token prediction of sensorimotor trajectories enables zero-shot humanoid walking and generalizes to unseen commands.
paper_ai_keywords: causal transformer autoregressive prediction sensorimotor trajectories modality-aligned prediction missing modalities simulated trajectories neural network policies model-based controllers motion capture data YouTube videos zero-shot learning real-world transfer generalization