FAST: Efficient Action Tokenization for Vision-Language-Action Models
Autoregressive sequence models, such as Transformer-based vision-language
action (VLA) policies, can be tremendously effective for capturing complex and
generalizable robotic behaviors. However, such models require us to choose a
tokenization of our continuous action signals, which determines how the
discrete symbols predicted by the model map to continuous robot actions. We
find that current approaches for robot action tokenization, based on simple
per-dimension, per-timestep binning schemes, typically perform poorly when
learning dexterous skills from high-frequency robot data. To address this
challenge, we propose a new compression-based tokenization scheme for robot
actions, based on the discrete cosine transform. Our tokenization approach,
Frequency-space Action Sequence Tokenization (FAST), enables us to train
autoregressive VLAs for highly dexterous and high-frequency tasks where
standard discretization methods fail completely. Based on FAST, we release
FAST+, a universal robot action tokenizer, trained on 1M real robot action
trajectories. It can be used as a black-box tokenizer for a wide range of robot
action sequences, with diverse action spaces and control frequencies. Finally,
we show that, when combined with the pi0 VLA, our method can scale to training
on 10k hours of robot data and match the performance of diffusion VLAs, while
reducing training time by up to 5x.
6789e918e1b3fda757de947f
Karl Pertsch
admin_assigned
2025-01-17T10:36:46.947Z
65d4c1ff29b4ac81c265e6e6
https://cdn-avatars.huggingface.co/v1/production/uploads/65d4c1ff29b4ac81c265e6e6/GXgs28okxGfpBdqhxov9-.png
Karl Pertsch
user
KarlP
6789e918e1b3fda757de9480
Kyle Stachowicz
admin_assigned
2025-01-17T10:36:59.830Z
6307eabda670ed10f9d2571f
/avatars/b0811b25ed4fb7e48dd380898049c764.svg
Kyle Stachowicz
user
kylestach
6789e918e1b3fda757de9481
Brian Ichter
admin_assigned
2025-01-17T10:37:05.229Z
633a26ab474cfeb1a864dc56
/avatars/cd40accd894fc78810fc0d5108f413e9.svg
Brian I
user
brianichter
6789e918e1b3fda757de9482
Danny Driess
admin_assigned
2025-01-17T10:37:10.829Z
67225875b46c703941fa7967
/avatars/7c89fbdd9a135210209bcd0cbfe7988a.svg
Danny Driess
user
dannydriess
6789e918e1b3fda757de9483
Suraj Nair
6789e918e1b3fda757de9484
Quan Vuong
6789e918e1b3fda757de9485
Oier Mees
admin_assigned
2025-01-17T10:37:30.435Z
663a7190d9dee283e3f56150
https://cdn-avatars.huggingface.co/v1/production/uploads/663a7190d9dee283e3f56150/zoYBdFIQPGWTS0R7bpg07.jpeg
Oier Mees
user
oier-mees
6789e918e1b3fda757de9486
Chelsea Finn
admin_assigned
2025-01-17T10:37:36.079Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
6789e918e1b3fda757de9487
Sergey Levine
admin_assigned
2025-01-17T10:37:41.213Z
665ce54120a307a3754849dd
/avatars/e698726e9be61dd50ce2efe372ed5dac.svg
Sergey Levine
user
svlevine
FAST: Efficient Action Tokenization for Vision-Language-Action Models
Autoregressive sequence models, such as Transformer-based vision-language
action (VLA) policies, can be tremendously effective for capturing complex and
generalizable robotic behaviors. However, such models require us to choose a
tokenization of our continuous action signals, which determines how the
discrete symbols predicted by the model map to continuous robot actions. We
find that current approaches for robot action tokenization, based on simple
per-dimension, per-timestep binning schemes, typically perform poorly when
learning dexterous skills from high-frequency robot data. To address this
challenge, we propose a new compression-based tokenization scheme for robot
actions, based on the discrete cosine transform. Our tokenization approach,
Frequency-space Action Sequence Tokenization (FAST), enables us to train
autoregressive VLAs for highly dexterous and high-frequency tasks where
standard discretization methods fail completely. Based on FAST, we release
FAST+, a universal robot action tokenizer, trained on 1M real robot action
trajectories. It can be used as a black-box tokenizer for a wide range of robot
action sequences, with diverse action spaces and control frequencies. Finally,
we show that, when combined with the pi0 VLA, our method can scale to training
on 10k hours of robot data and match the performance of diffusion VLAs, while
reducing training time by up to 5x.
A new tokenization method called FAST, based on the discrete cosine transform, enables training of autoregressive vision-language-action policies for high-frequency robotic tasks with improved performance and reduced training time.
autoregressive sequence models Transformer-based vision-language action policies tokenization continuous action signals discrete cosine transform Frequency-space Action Sequence Tokenization FAST FAST+ pi0 VLA diffusion VLAs