title: Octopus: Embodied Vision-Language Programmer from Environmental Feedback
summary: Large vision-language models (VLMs) have achieved substantial progress in
  multimodal perception and reasoning. Furthermore, when seamlessly integrated
  into an embodied agent, it signifies a crucial stride towards the creation of
  autonomous and context-aware systems capable of formulating plans and executing
  commands with precision. In this paper, we introduce Octopus, a novel VLM
  designed to proficiently decipher an agent's vision and textual task objectives
  and to formulate intricate action sequences and generate executable code. Our
  design allows the agent to adeptly handle a wide spectrum of tasks, ranging
  from mundane daily chores in simulators to sophisticated interactions in
  complex video games. Octopus is trained by leveraging GPT-4 to control an
  explorative agent to generate training data, i.e., action blueprints and the
  corresponding executable code, within our experimental environment called
  OctoVerse. We also collect the feedback that allows the enhanced training
  scheme of Reinforcement Learning with Environmental Feedback (RLEF). Through a
  series of experiments, we illuminate Octopus's functionality and present
  compelling results, and the proposed RLEF turns out to refine the agent's
  decision-making. By open-sourcing our model architecture, simulator, and
  dataset, we aspire to ignite further innovation and foster collaborative
  applications within the broader embodied AI community.
paper_authors._id: 6529471c019a20ed658f07d9
paper_authors.name: Jingkang Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-13T14:42:54.592Z
paper_authors._id: 62b5777f593a2c49da69dc02
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658152070753-62b5777f593a2c49da69dc02.jpeg
paper_authors.fullname: Jingkang Yang
paper_authors.type: user
paper_authors.user: Jingkang
paper_authors._id: 6529471c019a20ed658f07da
paper_authors.name: Yuhao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:50:15.190Z
paper_authors._id: 652965773a416e1f2173443b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
paper_authors.fullname: Yuhao Dong
paper_authors.type: user
paper_authors.user: THUdyh
paper_authors._id: 6529471c019a20ed658f07db
paper_authors.name: Shuai Liu
paper_authors._id: 6529471c019a20ed658f07dc
paper_authors.name: Bo Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:49:39.633Z
paper_authors._id: 62d3f7d84b0933c48f3cdd9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
paper_authors.fullname: Bo Li
paper_authors.type: user
paper_authors.user: luodian
paper_authors._id: 6529471c019a20ed658f07dd
paper_authors.name: Ziyue Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-16T07:24:45.666Z
paper_authors._id: 652961d24267f8c80220bbfb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0yCP-XYJu6PwTGkoyhKQB.jpeg
paper_authors.fullname: Ziyue Wang
paper_authors.type: user
paper_authors.user: annewzy
paper_authors._id: 6529471c019a20ed658f07de
paper_authors.name: Chencheng Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:42:07.445Z
paper_authors._id: 6470f5aad9360cd9d8e9310d
paper_authors.avatarUrl: /avatars/86efd78d7ff12bded166f0a5e3fe2763.svg
paper_authors.fullname: Jiang Chencheng
paper_authors.type: user
paper_authors.user: jchencheng
paper_authors._id: 6529471c019a20ed658f07df
paper_authors.name: Haoran Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-21T09:22:51.468Z
paper_authors._id: 652802a42949e6071c703550
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652802a42949e6071c703550/dQF73cLlAoSLtgi8JzbnI.jpeg
paper_authors.fullname: Haoran Tan
paper_authors.type: user
paper_authors.user: Tan0118
paper_authors._id: 6529471c019a20ed658f07e0
paper_authors.name: Jiamu Kang
paper_authors._id: 6529471c019a20ed658f07e1
paper_authors.name: Yuanhan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:39:27.007Z
paper_authors._id: 62a993d80472c0b7f94027df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
paper_authors.fullname: Zhang Yuanhan
paper_authors.type: user
paper_authors.user: ZhangYuanhan
paper_authors._id: 6529471c019a20ed658f07e2
paper_authors.name: Kaiyang Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T14:44:00.926Z
paper_authors._id: 62ac6656de8bfbb93094b8fd
paper_authors.avatarUrl: /avatars/1d8f445b6d5f9d43ddab81056bcb141e.svg
paper_authors.fullname: Kaiyang Zhou
paper_authors.type: user
paper_authors.user: kaiyangzhou
paper_authors._id: 6529471c019a20ed658f07e3
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T14:43:53.257Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: Octopus: Embodied Vision-Language Programmer from Environmental Feedback
paper_summary: Large vision-language models (VLMs) have achieved substantial progress in
  multimodal perception and reasoning. Furthermore, when seamlessly integrated
  into an embodied agent, it signifies a crucial stride towards the creation of
  autonomous and context-aware systems capable of formulating plans and executing
  commands with precision. In this paper, we introduce Octopus, a novel VLM
  designed to proficiently decipher an agent's vision and textual task objectives
  and to formulate intricate action sequences and generate executable code. Our
  design allows the agent to adeptly handle a wide spectrum of tasks, ranging
  from mundane daily chores in simulators to sophisticated interactions in
  complex video games. Octopus is trained by leveraging GPT-4 to control an
  explorative agent to generate training data, i.e., action blueprints and the
  corresponding executable code, within our experimental environment called
  OctoVerse. We also collect the feedback that allows the enhanced training
  scheme of Reinforcement Learning with Environmental Feedback (RLEF). Through a
  series of experiments, we illuminate Octopus's functionality and present
  compelling results, and the proposed RLEF turns out to refine the agent's
  decision-making. By open-sourcing our model architecture, simulator, and
  dataset, we aspire to ignite further innovation and foster collaborative
  applications within the broader embodied AI community.
paper_ai_keywords: large vision-language models (VLMs) multimodal perception embodied agent autonomous systems context-aware systems formulating plans executing commands Octopus vision textual task objectives action sequences executable code wide spectrum of tasks simulators complex video games GPT-4 explorative agent action blueprints Reinforcement Learning with Environmental Feedback (RLEF) OctoVerse