JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play
  Visual Games with Keyboards and Mouse
Recently, action-based decision-making in open-world environments has gained
significant attention. Visual Language Action (VLA) models, pretrained on
large-scale web datasets, have shown promise in decision-making tasks. However,
previous work has primarily focused on action post-training, often neglecting
enhancements to the foundational model itself. In response, we introduce a
novel approach, Act from Visual Language Post-Training, which refines Visual
Language Models (VLMs) through visual and linguistic guidance in a
self-supervised manner. This enhancement improves the models' capabilities in
world knowledge, visual recognition, and spatial grounding in open-world
environments. Following the above post-training paradigms, we obtain the first
VLA models in Minecraft that can follow human instructions on over 1k different
atomic tasks, including crafting, smelting, cooking, mining, and killing. Our
experiments demonstrate that post-training on non-trajectory tasks leads to a
significant 40% improvement over the best agent baseline on a diverse set of
atomic tasks. Furthermore, we demonstrate that our approach surpasses
traditional imitation learning-based policies in Minecraft, achieving
state-of-the-art performance. We have open-sourced the code, models, and
datasets to foster further research. The project page can be found in
https://craftjarvis.github.io/JarvisVLA.
67dcdc98e406e84ea880ccab
Muyao Li
67dcdc98e406e84ea880ccac
Zihao Wang
admin_assigned
2025-03-21T13:06:04.185Z
642e8c99c1b0f8e4e76bcaab
https://cdn-avatars.huggingface.co/v1/production/uploads/642e8c99c1b0f8e4e76bcaab/BOs9r0P9KyT9pEba9v0H4.png
Zihao Wang
user
zhwang4ai
67dcdc98e406e84ea880ccad
Kaichen He
admin_assigned
2025-03-21T13:05:06.674Z
6780ece9ff3623d9800b2183
/avatars/493e765bba03c115131ac16827c65d3e.svg
kaichen he
user
hkc20
67dcdc98e406e84ea880ccae
Xiaojian Ma
admin_assigned
2025-03-21T13:04:56.172Z
60dd0e36a15ddd7d2006d2e9
/avatars/8bd98177a79efbf295be8f6457683297.svg
Xiaojian Ma
user
jeasinema
67dcdc98e406e84ea880ccaf
Yitao Liang
admin_assigned
2025-03-21T13:04:49.708Z
64683a5776bb704aa14588b7
/avatars/e532756f52c5b95981470ace41a10556.svg
Yitao Liang
user
YitaoLiang
JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play
  Visual Games with Keyboards and Mouse
Recently, action-based decision-making in open-world environments has gained
significant attention. Visual Language Action (VLA) models, pretrained on
large-scale web datasets, have shown promise in decision-making tasks. However,
previous work has primarily focused on action post-training, often neglecting
enhancements to the foundational model itself. In response, we introduce a
novel approach, Act from Visual Language Post-Training, which refines Visual
Language Models (VLMs) through visual and linguistic guidance in a
self-supervised manner. This enhancement improves the models' capabilities in
world knowledge, visual recognition, and spatial grounding in open-world
environments. Following the above post-training paradigms, we obtain the first
VLA models in Minecraft that can follow human instructions on over 1k different
atomic tasks, including crafting, smelting, cooking, mining, and killing. Our
experiments demonstrate that post-training on non-trajectory tasks leads to a
significant 40% improvement over the best agent baseline on a diverse set of
atomic tasks. Furthermore, we demonstrate that our approach surpasses
traditional imitation learning-based policies in Minecraft, achieving
state-of-the-art performance. We have open-sourced the code, models, and
datasets to foster further research. The project page can be found in
https://craftjarvis.github.io/JarvisVLA.
A novel self-supervised post-training method for Visual Language Models (VLMs) enhances their performance in action-based decision-making tasks, achieving state-of-the-art results in Minecraft.
Visual Language Action (VLA) models Visual Language Models (VLMs) self-supervised learning world knowledge visual recognition spatial grounding atomic tasks imitation learning Minecraft