title: OmniJARVIS: Unified Vision-Language-Action Tokenization Enables
    Open-World Instruction Following Agents
summary: We present OmniJARVIS, a novel Vision-Language-Action (VLA) model for
  open-world instruction-following agents in open-world Minecraft. Compared to
  prior works that either emit textual goals to separate controllers or produce
  the control command directly, OmniJARVIS seeks a different path to ensure both
  strong reasoning and efficient decision-making capabilities via unified
  tokenization of multimodal interaction data. First, we introduce a
  self-supervised approach to learn a behavior encoder that produces discretized
  tokens for behavior trajectories tau = {o_0, a_0, dots} and an
  imitation learning (IL) policy decoder conditioned on these tokens. These
  additional behavior tokens will be augmented to the vocabulary of pretrained
  Multimodal Language Models (MLMs). With this encoder, we then pack long-term
  multimodal interactions involving task instructions, memories, thoughts,
  observations, textual responses, behavior trajectories, etc. into unified token
  sequences and model them with autoregressive transformers. Thanks to the
  semantically meaningful behavior tokens, the resulting VLA model, OmniJARVIS,
  can reason (by producing chain-of-thoughts), plan, answer questions, and act
  (by producing behavior tokens for the IL policy decoder). OmniJARVIS
  demonstrates excellent performances on a comprehensive collection of atomic,
  programmatic, and open-ended tasks in open-world Minecraft. Our analysis
  further unveils the crucial design principles in interaction data formation,
  unified tokenization, and its scaling potentials.
paper_authors._id: 6683bcf921fcc56c785f6d0a
paper_authors.name: Zihao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:05:06.581Z
paper_authors._id: 642e8c99c1b0f8e4e76bcaab
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642e8c99c1b0f8e4e76bcaab/BOs9r0P9KyT9pEba9v0H4.png
paper_authors.fullname: Zihao Wang
paper_authors.type: user
paper_authors.user: zhwang4ai
paper_authors._id: 6683bcf921fcc56c785f6d0b
paper_authors.name: Shaofei Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-03T07:41:07.815Z
paper_authors._id: 6578459d62d3ac1817ed79fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6578459d62d3ac1817ed79fe/AXDJuwLUoEOb4Fj3U0Xxo.jpeg
paper_authors.fullname: Shaofei Cai
paper_authors.type: user
paper_authors.user: phython96
paper_authors._id: 6683bcf921fcc56c785f6d0c
paper_authors.name: Zhancun Mu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T07:01:23.829Z
paper_authors._id: 648f0c64ddc0620d54e199e9
paper_authors.avatarUrl: /avatars/ff3fd6e11d451b03f82183e53fc48613.svg
paper_authors.fullname: ZhancunMu
paper_authors.type: user
paper_authors.user: Zhancun
paper_authors._id: 6683bcf921fcc56c785f6d0d
paper_authors.name: Haowei Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T07:01:26.143Z
paper_authors._id: 63453f02a05b51f7ded3c579
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/u7p-wTj8zyeWpSFRI8Go0.png
paper_authors.fullname: Andy Lin
paper_authors.type: user
paper_authors.user: pkuHaowei
paper_authors._id: 6683bcf921fcc56c785f6d0e
paper_authors.name: Ceyao Zhang
paper_authors._id: 6683bcf921fcc56c785f6d0f
paper_authors.name: Xuejie Liu
paper_authors._id: 6683bcf921fcc56c785f6d10
paper_authors.name: Qing Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-03T08:26:12.303Z
paper_authors._id: 6455f8c2c8f569b995d603c9
paper_authors.avatarUrl: /avatars/3cb2cad3ab123887c270234bb6a4ca43.svg
paper_authors.fullname: Qing Li
paper_authors.type: user
paper_authors.user: li-qing
paper_authors._id: 6683bcf921fcc56c785f6d11
paper_authors.name: Anji Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T15:16:15.361Z
paper_authors._id: 66757955b3882fd587d5f363
paper_authors.avatarUrl: /avatars/41fdfa19057ad3ea5ece29b94f163218.svg
paper_authors.fullname: Anji Liu
paper_authors.type: user
paper_authors.user: anjiliu
paper_authors._id: 6683bcf921fcc56c785f6d12
paper_authors.name: Xiaojian Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T08:17:50.054Z
paper_authors._id: 60dd0e36a15ddd7d2006d2e9
paper_authors.avatarUrl: /avatars/8bd98177a79efbf295be8f6457683297.svg
paper_authors.fullname: Xiaojian Ma
paper_authors.type: user
paper_authors.user: jeasinema
paper_authors._id: 6683bcf921fcc56c785f6d13
paper_authors.name: Yitao Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T07:38:35.013Z
paper_authors._id: 64683a5776bb704aa14588b7
paper_authors.avatarUrl: /avatars/e532756f52c5b95981470ace41a10556.svg
paper_authors.fullname: Yitao Liang
paper_authors.type: user
paper_authors.user: YitaoLiang
paper_title: OmniJARVIS: Unified Vision-Language-Action Tokenization Enables
    Open-World Instruction Following Agents
paper_summary: We present OmniJARVIS, a novel Vision-Language-Action (VLA) model for
  open-world instruction-following agents in open-world Minecraft. Compared to
  prior works that either emit textual goals to separate controllers or produce
  the control command directly, OmniJARVIS seeks a different path to ensure both
  strong reasoning and efficient decision-making capabilities via unified
  tokenization of multimodal interaction data. First, we introduce a
  self-supervised approach to learn a behavior encoder that produces discretized
  tokens for behavior trajectories tau = {o_0, a_0, dots} and an
  imitation learning (IL) policy decoder conditioned on these tokens. These
  additional behavior tokens will be augmented to the vocabulary of pretrained
  Multimodal Language Models (MLMs). With this encoder, we then pack long-term
  multimodal interactions involving task instructions, memories, thoughts,
  observations, textual responses, behavior trajectories, etc. into unified token
  sequences and model them with autoregressive transformers. Thanks to the
  semantically meaningful behavior tokens, the resulting VLA model, OmniJARVIS,
  can reason (by producing chain-of-thoughts), plan, answer questions, and act
  (by producing behavior tokens for the IL policy decoder). OmniJARVIS
  demonstrates excellent performances on a comprehensive collection of atomic,
  programmatic, and open-ended tasks in open-world Minecraft. Our analysis
  further unveils the crucial design principles in interaction data formation,
  unified tokenization, and its scaling potentials.
paper_ai_summary: OmniJARVIS, a Vision-Language-Action model for Minecraft, uses unified tokenization of multimodal interaction data with a behavior encoder and autoregressive transformers to perform reasoning, planning, question answering, and action execution.
paper_ai_keywords: self-supervised learning behavior encoder imitation learning policy decoder Multimodal Language Models unified tokenization autoregressive transformers chain-of-thoughts