title: JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play
    Visual Games with Keyboards and Mouse
summary: Recently, action-based decision-making in open-world environments has gained
  significant attention. Visual Language Action (VLA) models, pretrained on
  large-scale web datasets, have shown promise in decision-making tasks. However,
  previous work has primarily focused on action post-training, often neglecting
  enhancements to the foundational model itself. In response, we introduce a
  novel approach, Act from Visual Language Post-Training, which refines Visual
  Language Models (VLMs) through visual and linguistic guidance in a
  self-supervised manner. This enhancement improves the models' capabilities in
  world knowledge, visual recognition, and spatial grounding in open-world
  environments. Following the above post-training paradigms, we obtain the first
  VLA models in Minecraft that can follow human instructions on over 1k different
  atomic tasks, including crafting, smelting, cooking, mining, and killing. Our
  experiments demonstrate that post-training on non-trajectory tasks leads to a
  significant 40% improvement over the best agent baseline on a diverse set of
  atomic tasks. Furthermore, we demonstrate that our approach surpasses
  traditional imitation learning-based policies in Minecraft, achieving
  state-of-the-art performance. We have open-sourced the code, models, and
  datasets to foster further research. The project page can be found in
  https://craftjarvis.github.io/JarvisVLA.
paper_authors._id: 67dcdc98e406e84ea880ccab
paper_authors.name: Muyao Li
paper_authors._id: 67dcdc98e406e84ea880ccac
paper_authors.name: Zihao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:06:04.185Z
paper_authors._id: 642e8c99c1b0f8e4e76bcaab
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642e8c99c1b0f8e4e76bcaab/BOs9r0P9KyT9pEba9v0H4.png
paper_authors.fullname: Zihao Wang
paper_authors.type: user
paper_authors.user: zhwang4ai
paper_authors._id: 67dcdc98e406e84ea880ccad
paper_authors.name: Kaichen He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:05:06.674Z
paper_authors._id: 6780ece9ff3623d9800b2183
paper_authors.avatarUrl: /avatars/493e765bba03c115131ac16827c65d3e.svg
paper_authors.fullname: kaichen he
paper_authors.type: user
paper_authors.user: hkc20
paper_authors._id: 67dcdc98e406e84ea880ccae
paper_authors.name: Xiaojian Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:04:56.172Z
paper_authors._id: 60dd0e36a15ddd7d2006d2e9
paper_authors.avatarUrl: /avatars/8bd98177a79efbf295be8f6457683297.svg
paper_authors.fullname: Xiaojian Ma
paper_authors.type: user
paper_authors.user: jeasinema
paper_authors._id: 67dcdc98e406e84ea880ccaf
paper_authors.name: Yitao Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:04:49.708Z
paper_authors._id: 64683a5776bb704aa14588b7
paper_authors.avatarUrl: /avatars/e532756f52c5b95981470ace41a10556.svg
paper_authors.fullname: Yitao Liang
paper_authors.type: user
paper_authors.user: YitaoLiang
paper_title: JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play
    Visual Games with Keyboards and Mouse
paper_summary: Recently, action-based decision-making in open-world environments has gained
  significant attention. Visual Language Action (VLA) models, pretrained on
  large-scale web datasets, have shown promise in decision-making tasks. However,
  previous work has primarily focused on action post-training, often neglecting
  enhancements to the foundational model itself. In response, we introduce a
  novel approach, Act from Visual Language Post-Training, which refines Visual
  Language Models (VLMs) through visual and linguistic guidance in a
  self-supervised manner. This enhancement improves the models' capabilities in
  world knowledge, visual recognition, and spatial grounding in open-world
  environments. Following the above post-training paradigms, we obtain the first
  VLA models in Minecraft that can follow human instructions on over 1k different
  atomic tasks, including crafting, smelting, cooking, mining, and killing. Our
  experiments demonstrate that post-training on non-trajectory tasks leads to a
  significant 40% improvement over the best agent baseline on a diverse set of
  atomic tasks. Furthermore, we demonstrate that our approach surpasses
  traditional imitation learning-based policies in Minecraft, achieving
  state-of-the-art performance. We have open-sourced the code, models, and
  datasets to foster further research. The project page can be found in
  https://craftjarvis.github.io/JarvisVLA.
paper_ai_summary: A novel self-supervised post-training method for Visual Language Models (VLMs) enhances their performance in action-based decision-making tasks, achieving state-of-the-art results in Minecraft.
paper_ai_keywords: Visual Language Action (VLA) models Visual Language Models (VLMs) self-supervised learning world knowledge visual recognition spatial grounding atomic tasks imitation learning Minecraft