Humanoid Locomotion as Next Token Prediction
We cast real-world humanoid control as a next token prediction problem, akin
to predicting the next word in language. Our model is a causal transformer
trained via autoregressive prediction of sensorimotor trajectories. To account
for the multi-modal nature of the data, we perform prediction in a
modality-aligned way, and for each input token predict the next token from the
same modality. This general formulation enables us to leverage data with
missing modalities, like video trajectories without actions. We train our model
on a collection of simulated trajectories coming from prior neural network
policies, model-based controllers, motion capture data, and YouTube videos of
humans. We show that our model enables a full-sized humanoid to walk in San
Francisco zero-shot. Our model can transfer to the real world even when trained
on only 27 hours of walking data, and can generalize to commands not seen
during training like walking backward. These findings suggest a promising path
toward learning challenging real-world control tasks by generative modeling of
sensorimotor trajectories.
65e16d5403a248cceb66bfd0
Ilija Radosavovic
65e16d5403a248cceb66bfd1
Bike Zhang
65e16d5403a248cceb66bfd2
Baifeng Shi
claimed_verified
2024-03-01T09:25:37.870Z
649004218f7cbbc94c782db6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
Baifeng Shi
user
bfshi
65e16d5403a248cceb66bfd3
Jathushan Rajasegaran
admin_assigned
2024-03-01T10:37:30.024Z
63895b3a43d8b0797a0d0406
/avatars/04d9952dff70f85d4e481ec80ae818cb.svg
Jathushan Rajasegaran
user
brjathu
65e16d5403a248cceb66bfd4
Sarthak Kamat
claimed_verified
2024-12-30T19:41:28.529Z
676edf92c1a36ee523b219c5
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_nxPANpjOkUXQxrQhV-q7.png
Sarthak Kamat
user
sarthakkamat
65e16d5403a248cceb66bfd5
Trevor Darrell
admin_assigned
2024-03-01T10:37:41.172Z
64cbdf02f103036e23d1c7f3
/avatars/496069463900dea20929b57381182d39.svg
Trevor Darrell
user
trevordarrell
65e16d5403a248cceb66bfd6
Koushil Sreenath
65e16d5403a248cceb66bfd7
Jitendra Malik
admin_assigned
2024-03-01T10:38:10.533Z
65369a95605a07338de78ab0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
Jitendra Malik 
user
jitendra1995
Humanoid Locomotion as Next Token Prediction
We cast real-world humanoid control as a next token prediction problem, akin
to predicting the next word in language. Our model is a causal transformer
trained via autoregressive prediction of sensorimotor trajectories. To account
for the multi-modal nature of the data, we perform prediction in a
modality-aligned way, and for each input token predict the next token from the
same modality. This general formulation enables us to leverage data with
missing modalities, like video trajectories without actions. We train our model
on a collection of simulated trajectories coming from prior neural network
policies, model-based controllers, motion capture data, and YouTube videos of
humans. We show that our model enables a full-sized humanoid to walk in San
Francisco zero-shot. Our model can transfer to the real world even when trained
on only 27 hours of walking data, and can generalize to commands not seen
during training like walking backward. These findings suggest a promising path
toward learning challenging real-world control tasks by generative modeling of
sensorimotor trajectories.
A causal transformer for next token prediction of sensorimotor trajectories enables zero-shot humanoid walking and generalizes to unseen commands.
causal transformer autoregressive prediction sensorimotor trajectories modality-aligned prediction missing modalities simulated trajectories neural network policies model-based controllers motion capture data YouTube videos zero-shot learning real-world transfer generalization