ROCKET-1: Master Open-World Interaction with Visual-Temporal Context
  Prompting
Vision-language models (VLMs) have excelled in multimodal tasks, but adapting
them to embodied decision-making in open-world environments presents
challenges. A key issue is the difficulty in smoothly connecting individual
entities in low-level observations with abstract concepts required for
planning. A common approach to address this problem is through the use of
hierarchical agents, where VLMs serve as high-level reasoners that break down
tasks into executable sub-tasks, typically specified using language and
imagined observations. However, language often fails to effectively convey
spatial information, while generating future images with sufficient accuracy
remains challenging. To address these limitations, we propose visual-temporal
context prompting, a novel communication protocol between VLMs and policy
models. This protocol leverages object segmentation from both past and present
observations to guide policy-environment interactions. Using this approach, we
train ROCKET-1, a low-level policy that predicts actions based on concatenated
visual observations and segmentation masks, with real-time object tracking
provided by SAM-2. Our method unlocks the full potential of VLMs
visual-language reasoning abilities, enabling them to solve complex creative
tasks, especially those heavily reliant on spatial understanding. Experiments
in Minecraft demonstrate that our approach allows agents to accomplish
previously unattainable tasks, highlighting the effectiveness of
visual-temporal context prompting in embodied decision-making. Codes and demos
will be available on the project page: https://craftjarvis.github.io/ROCKET-1.
6719e7d84dfd79aa9f3f6c69
Shaofei Cai
claimed_verified
2024-10-28T11:25:08.953Z
6578459d62d3ac1817ed79fe
https://cdn-avatars.huggingface.co/v1/production/uploads/6578459d62d3ac1817ed79fe/AXDJuwLUoEOb4Fj3U0Xxo.jpeg
Shaofei Cai
user
phython96
6719e7d84dfd79aa9f3f6c6a
Zihao Wang
claimed_verified
2024-12-17T08:04:45.236Z
642e8c99c1b0f8e4e76bcaab
https://cdn-avatars.huggingface.co/v1/production/uploads/642e8c99c1b0f8e4e76bcaab/BOs9r0P9KyT9pEba9v0H4.png
Zihao Wang
user
zhwang4ai
6719e7d84dfd79aa9f3f6c6b
Kewei Lian
claimed_verified
2024-10-28T11:24:52.514Z
64c232e577655fcf3ff06082
/avatars/5fc3ae34273c96f761a1eb9d336c64b9.svg
Kewei Lian
user
kevinLian
6719e7d84dfd79aa9f3f6c6c
Zhancun Mu
admin_assigned
2024-10-28T15:17:23.469Z
648f0c64ddc0620d54e199e9
/avatars/ff3fd6e11d451b03f82183e53fc48613.svg
ZhancunMu
user
Zhancun
6719e7d84dfd79aa9f3f6c6d
Xiaojian Ma
admin_assigned
2024-10-28T15:17:12.954Z
60dd0e36a15ddd7d2006d2e9
/avatars/8bd98177a79efbf295be8f6457683297.svg
Xiaojian Ma
user
jeasinema
6719e7d84dfd79aa9f3f6c6e
Anji Liu
claimed_verified
2024-10-28T15:16:10.809Z
66757955b3882fd587d5f363
/avatars/41fdfa19057ad3ea5ece29b94f163218.svg
Anji Liu
user
anjiliu
6719e7d84dfd79aa9f3f6c6f
Yitao Liang
claimed_verified
2025-04-09T07:38:25.072Z
64683a5776bb704aa14588b7
/avatars/e532756f52c5b95981470ace41a10556.svg
Yitao Liang
user
YitaoLiang
ROCKET-1: Master Open-World Interaction with Visual-Temporal Context
  Prompting
Vision-language models (VLMs) have excelled in multimodal tasks, but adapting
them to embodied decision-making in open-world environments presents
challenges. A key issue is the difficulty in smoothly connecting individual
entities in low-level observations with abstract concepts required for
planning. A common approach to address this problem is through the use of
hierarchical agents, where VLMs serve as high-level reasoners that break down
tasks into executable sub-tasks, typically specified using language and
imagined observations. However, language often fails to effectively convey
spatial information, while generating future images with sufficient accuracy
remains challenging. To address these limitations, we propose visual-temporal
context prompting, a novel communication protocol between VLMs and policy
models. This protocol leverages object segmentation from both past and present
observations to guide policy-environment interactions. Using this approach, we
train ROCKET-1, a low-level policy that predicts actions based on concatenated
visual observations and segmentation masks, with real-time object tracking
provided by SAM-2. Our method unlocks the full potential of VLMs
visual-language reasoning abilities, enabling them to solve complex creative
tasks, especially those heavily reliant on spatial understanding. Experiments
in Minecraft demonstrate that our approach allows agents to accomplish
previously unattainable tasks, highlighting the effectiveness of
visual-temporal context prompting in embodied decision-making. Codes and demos
will be available on the project page: https://craftjarvis.github.io/ROCKET-1.
Visual-temporal context prompting integrates object segmentation and visual observations to enhance VLMs in embodied decision-making, enabling complex spatial tasks.
vision-language models hierarchical agents visual-temporal context prompting object segmentation policy models real-time object tracking SAM-2 ROCKET-1 Minecraft