OpenVLA: An Open-Source Vision-Language-Action Model
Large policies pretrained on a combination of Internet-scale vision-language
data and diverse robot demonstrations have the potential to change how we teach
robots new skills: rather than training new behaviors from scratch, we can
fine-tune such vision-language-action (VLA) models to obtain robust,
generalizable policies for visuomotor control. Yet, widespread adoption of VLAs
for robotics has been challenging as 1) existing VLAs are largely closed and
inaccessible to the public, and 2) prior work fails to explore methods for
efficiently fine-tuning VLAs for new tasks, a key component for adoption.
Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source
VLA trained on a diverse collection of 970k real-world robot demonstrations.
OpenVLA builds on a Llama 2 language model combined with a visual encoder that
fuses pretrained features from DINOv2 and SigLIP. As a product of the added
data diversity and new model components, OpenVLA demonstrates strong results
for generalist manipulation, outperforming closed models such as RT-2-X (55B)
by 16.5% in absolute task success rate across 29 tasks and multiple robot
embodiments, with 7x fewer parameters. We further show that we can effectively
fine-tune OpenVLA for new settings, with especially strong generalization
results in multi-task environments involving multiple objects and strong
language grounding abilities, and outperform expressive from-scratch imitation
learning methods such as Diffusion Policy by 20.4%. We also explore compute
efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned
on consumer GPUs via modern low-rank adaptation methods and served efficiently
via quantization without a hit to downstream success rate. Finally, we release
model checkpoints, fine-tuning notebooks, and our PyTorch codebase with
built-in support for training VLAs at scale on Open X-Embodiment datasets.
666b9d34cf416fdc65f87643
Moo Jin Kim
admin_assigned
2024-06-14T07:20:40.563Z
64b017833ffa69983bb8ed37
https://cdn-avatars.huggingface.co/v1/production/uploads/64b017833ffa69983bb8ed37/gdaFjlaJDDPtfH84b75RU.jpeg
Moo Jin Kim
user
moojink
666b9d34cf416fdc65f87644
Karl Pertsch
admin_assigned
2024-06-14T07:20:48.705Z
65d4c1ff29b4ac81c265e6e6
https://cdn-avatars.huggingface.co/v1/production/uploads/65d4c1ff29b4ac81c265e6e6/GXgs28okxGfpBdqhxov9-.png
Karl Pertsch
user
KarlP
666b9d34cf416fdc65f87645
Siddharth Karamcheti
admin_assigned
2024-06-14T07:20:55.211Z
6150b090d84cf0532aa1764b
https://cdn-avatars.huggingface.co/v1/production/uploads/1634666324094-6150b090d84cf0532aa1764b.jpeg
Siddharth Karamcheti
user
skaramcheti
666b9d34cf416fdc65f87646
Ted Xiao
admin_assigned
2024-06-14T07:21:02.137Z
64246a35089d5fae56b4a584
/avatars/bc2f1f1135dbc3460ae30b0e7a0cf39b.svg
xiao
user
tedxiao
666b9d34cf416fdc65f87647
Ashwin Balakrishna
admin_assigned
2024-06-14T10:04:21.971Z
653ad464170f0a2a9202dbe3
/avatars/dd48cf05b946663dcb927545cd8edddc.svg
Ashwin Balakrishna
user
abalakrishnaTRI
666b9d34cf416fdc65f87648
Suraj Nair
admin_assigned
2024-06-14T10:04:40.855Z
624f0e38c8b211c193d325dd
/avatars/370b7b21bd1bc284d3f48c0c40dfa026.svg
Suraj Nair
user
surajnair
666b9d34cf416fdc65f87649
Rafael Rafailov
admin_assigned
2024-06-14T10:04:47.337Z
64701c98d742e9ef6522598f
/avatars/df3a52bd4bc374d7cb4061f449f75c07.svg
Rafael Rafailov
user
rmrafailov
666b9d34cf416fdc65f8764a
Ethan Foster
666b9d34cf416fdc65f8764b
Grace Lam
666b9d34cf416fdc65f8764c
Pannag Sanketi
claimed_verified
2024-08-09T07:48:52.353Z
66b56f73fc66a466e6104bfb
/avatars/74855aaf580144fafead4b93d6a24a46.svg
Pannag Sanketi
user
prsanky
666b9d34cf416fdc65f8764d
Quan Vuong
admin_assigned
2024-06-14T10:05:19.625Z
63e1d6687fbb6ae4d4f4d980
/avatars/4959319e69d69e03a31ff4be326e602a.svg
Quan Vuong
user
quintonx
666b9d34cf416fdc65f8764e
Thomas Kollar
admin_assigned
2024-06-14T10:05:34.367Z
64358e662d0ed796668cb438
/avatars/6f86547d8691e061e289dd906deedfdb.svg
Thomas Kollar
user
thomas-kollar
666b9d34cf416fdc65f8764f
Benjamin Burchfiel
666b9d34cf416fdc65f87650
Russ Tedrake
666b9d34cf416fdc65f87651
Dorsa Sadigh
666b9d34cf416fdc65f87652
Sergey Levine
admin_assigned
2024-06-14T10:05:50.110Z
665ce54120a307a3754849dd
/avatars/e698726e9be61dd50ce2efe372ed5dac.svg
Sergey Levine
user
svlevine
666b9d34cf416fdc65f87653
Percy Liang
admin_assigned
2024-06-14T10:05:55.774Z
6409651b9e9f790c905b2335
/avatars/1fb8c80b60f21f65a0a027319101f236.svg
Percy Liang
user
percyliang
666b9d34cf416fdc65f87654
Chelsea Finn
admin_assigned
2024-06-14T10:06:01.500Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
OpenVLA: An Open-Source Vision-Language-Action Model
Large policies pretrained on a combination of Internet-scale vision-language
data and diverse robot demonstrations have the potential to change how we teach
robots new skills: rather than training new behaviors from scratch, we can
fine-tune such vision-language-action (VLA) models to obtain robust,
generalizable policies for visuomotor control. Yet, widespread adoption of VLAs
for robotics has been challenging as 1) existing VLAs are largely closed and
inaccessible to the public, and 2) prior work fails to explore methods for
efficiently fine-tuning VLAs for new tasks, a key component for adoption.
Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source
VLA trained on a diverse collection of 970k real-world robot demonstrations.
OpenVLA builds on a Llama 2 language model combined with a visual encoder that
fuses pretrained features from DINOv2 and SigLIP. As a product of the added
data diversity and new model components, OpenVLA demonstrates strong results
for generalist manipulation, outperforming closed models such as RT-2-X (55B)
by 16.5% in absolute task success rate across 29 tasks and multiple robot
embodiments, with 7x fewer parameters. We further show that we can effectively
fine-tune OpenVLA for new settings, with especially strong generalization
results in multi-task environments involving multiple objects and strong
language grounding abilities, and outperform expressive from-scratch imitation
learning methods such as Diffusion Policy by 20.4%. We also explore compute
efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned
on consumer GPUs via modern low-rank adaptation methods and served efficiently
via quantization without a hit to downstream success rate. Finally, we release
model checkpoints, fine-tuning notebooks, and our PyTorch codebase with
built-in support for training VLAs at scale on Open X-Embodiment datasets.
OpenVLA, a 7B-parameter open-source vision-language-action model, demonstrates strong performance in generalist manipulation and efficient fine-tuning for new tasks, outperforming larger closed models and from-scratch imitation learning methods.
Llama 2 DINOv2 SigLIP generalist manipulation RT-2-X Diffusion Policy low-rank adaptation quantization Open X-Embodiment datasets