title: ROCKET-1: Master Open-World Interaction with Visual-Temporal Context
    Prompting
summary: Vision-language models (VLMs) have excelled in multimodal tasks, but adapting
  them to embodied decision-making in open-world environments presents
  challenges. A key issue is the difficulty in smoothly connecting individual
  entities in low-level observations with abstract concepts required for
  planning. A common approach to address this problem is through the use of
  hierarchical agents, where VLMs serve as high-level reasoners that break down
  tasks into executable sub-tasks, typically specified using language and
  imagined observations. However, language often fails to effectively convey
  spatial information, while generating future images with sufficient accuracy
  remains challenging. To address these limitations, we propose visual-temporal
  context prompting, a novel communication protocol between VLMs and policy
  models. This protocol leverages object segmentation from both past and present
  observations to guide policy-environment interactions. Using this approach, we
  train ROCKET-1, a low-level policy that predicts actions based on concatenated
  visual observations and segmentation masks, with real-time object tracking
  provided by SAM-2. Our method unlocks the full potential of VLMs
  visual-language reasoning abilities, enabling them to solve complex creative
  tasks, especially those heavily reliant on spatial understanding. Experiments
  in Minecraft demonstrate that our approach allows agents to accomplish
  previously unattainable tasks, highlighting the effectiveness of
  visual-temporal context prompting in embodied decision-making. Codes and demos
  will be available on the project page: https://craftjarvis.github.io/ROCKET-1.
paper_authors._id: 6719e7d84dfd79aa9f3f6c69
paper_authors.name: Shaofei Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:25:08.953Z
paper_authors._id: 6578459d62d3ac1817ed79fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6578459d62d3ac1817ed79fe/AXDJuwLUoEOb4Fj3U0Xxo.jpeg
paper_authors.fullname: Shaofei Cai
paper_authors.type: user
paper_authors.user: phython96
paper_authors._id: 6719e7d84dfd79aa9f3f6c6a
paper_authors.name: Zihao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:04:45.236Z
paper_authors._id: 642e8c99c1b0f8e4e76bcaab
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642e8c99c1b0f8e4e76bcaab/BOs9r0P9KyT9pEba9v0H4.png
paper_authors.fullname: Zihao Wang
paper_authors.type: user
paper_authors.user: zhwang4ai
paper_authors._id: 6719e7d84dfd79aa9f3f6c6b
paper_authors.name: Kewei Lian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:24:52.514Z
paper_authors._id: 64c232e577655fcf3ff06082
paper_authors.avatarUrl: /avatars/5fc3ae34273c96f761a1eb9d336c64b9.svg
paper_authors.fullname: Kewei Lian
paper_authors.type: user
paper_authors.user: kevinLian
paper_authors._id: 6719e7d84dfd79aa9f3f6c6c
paper_authors.name: Zhancun Mu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:17:23.469Z
paper_authors._id: 648f0c64ddc0620d54e199e9
paper_authors.avatarUrl: /avatars/ff3fd6e11d451b03f82183e53fc48613.svg
paper_authors.fullname: ZhancunMu
paper_authors.type: user
paper_authors.user: Zhancun
paper_authors._id: 6719e7d84dfd79aa9f3f6c6d
paper_authors.name: Xiaojian Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:17:12.954Z
paper_authors._id: 60dd0e36a15ddd7d2006d2e9
paper_authors.avatarUrl: /avatars/8bd98177a79efbf295be8f6457683297.svg
paper_authors.fullname: Xiaojian Ma
paper_authors.type: user
paper_authors.user: jeasinema
paper_authors._id: 6719e7d84dfd79aa9f3f6c6e
paper_authors.name: Anji Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T15:16:10.809Z
paper_authors._id: 66757955b3882fd587d5f363
paper_authors.avatarUrl: /avatars/41fdfa19057ad3ea5ece29b94f163218.svg
paper_authors.fullname: Anji Liu
paper_authors.type: user
paper_authors.user: anjiliu
paper_authors._id: 6719e7d84dfd79aa9f3f6c6f
paper_authors.name: Yitao Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T07:38:25.072Z
paper_authors._id: 64683a5776bb704aa14588b7
paper_authors.avatarUrl: /avatars/e532756f52c5b95981470ace41a10556.svg
paper_authors.fullname: Yitao Liang
paper_authors.type: user
paper_authors.user: YitaoLiang
paper_title: ROCKET-1: Master Open-World Interaction with Visual-Temporal Context
    Prompting
paper_summary: Vision-language models (VLMs) have excelled in multimodal tasks, but adapting
  them to embodied decision-making in open-world environments presents
  challenges. A key issue is the difficulty in smoothly connecting individual
  entities in low-level observations with abstract concepts required for
  planning. A common approach to address this problem is through the use of
  hierarchical agents, where VLMs serve as high-level reasoners that break down
  tasks into executable sub-tasks, typically specified using language and
  imagined observations. However, language often fails to effectively convey
  spatial information, while generating future images with sufficient accuracy
  remains challenging. To address these limitations, we propose visual-temporal
  context prompting, a novel communication protocol between VLMs and policy
  models. This protocol leverages object segmentation from both past and present
  observations to guide policy-environment interactions. Using this approach, we
  train ROCKET-1, a low-level policy that predicts actions based on concatenated
  visual observations and segmentation masks, with real-time object tracking
  provided by SAM-2. Our method unlocks the full potential of VLMs
  visual-language reasoning abilities, enabling them to solve complex creative
  tasks, especially those heavily reliant on spatial understanding. Experiments
  in Minecraft demonstrate that our approach allows agents to accomplish
  previously unattainable tasks, highlighting the effectiveness of
  visual-temporal context prompting in embodied decision-making. Codes and demos
  will be available on the project page: https://craftjarvis.github.io/ROCKET-1.
paper_ai_summary: Visual-temporal context prompting integrates object segmentation and visual observations to enhance VLMs in embodied decision-making, enabling complex spatial tasks.
paper_ai_keywords: vision-language models hierarchical agents visual-temporal context prompting object segmentation policy models real-time object tracking SAM-2 ROCKET-1 Minecraft