Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in
  Long-Horizon Tasks
Building a general-purpose agent is a long-standing vision in the field of
artificial intelligence. Existing agents have made remarkable progress in many
domains, yet they still struggle to complete long-horizon tasks in an open
world. We attribute this to the lack of necessary world knowledge and
multimodal experience that can guide agents through a variety of long-horizon
tasks. In this paper, we propose a Hybrid Multimodal Memory module to address
the above challenges. It 1) transforms knowledge into Hierarchical Directed
Knowledge Graph that allows agents to explicitly represent and learn world
knowledge, and 2) summarises historical information into Abstracted Multimodal
Experience Pool that provide agents with rich references for in-context
learning. On top of the Hybrid Multimodal Memory module, a multimodal agent,
Optimus-1, is constructed with dedicated Knowledge-guided Planner and
Experience-Driven Reflector, contributing to a better planning and reflection
in the face of long-horizon tasks in Minecraft. Extensive experimental results
show that Optimus-1 significantly outperforms all existing agents on
challenging long-horizon task benchmarks, and exhibits near human-level
performance on many tasks. In addition, we introduce various Multimodal Large
Language Models (MLLMs) as the backbone of Optimus-1. Experimental results show
that Optimus-1 exhibits strong generalization with the help of the Hybrid
Multimodal Memory module, outperforming the GPT-4V baseline on many tasks.
66b426278a1be1e8acc2a716
Zaijing Li
claimed_verified
2024-08-08T10:13:09.178Z
66b45fe75d0ac130d7d82764
/avatars/09253f41f82c533b36199f82620cd075.svg
Zaijing Li
user
dawn0815
66b426278a1be1e8acc2a717
Yuquan Xie
admin_assigned
2024-08-08T10:13:23.374Z
64b8e4760de7289de37450a6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/q0q7LctHeuD-qtTZOtOP7.jpeg
Yuquan Xie
user
xieyuquan
66b426278a1be1e8acc2a718
Rui Shao
admin_assigned
2024-08-08T10:13:33.784Z
64cf57956f107411da90ee42
https://cdn-avatars.huggingface.co/v1/production/uploads/64cf57956f107411da90ee42/CQc3W9trxFDJiN2ZAcaxB.jpeg
Rui Shao
user
rshaojimmy
66b426278a1be1e8acc2a719
Gongwei Chen
claimed_verified
2024-08-08T10:52:29.117Z
66b4a3575fa7debf39254e5c
/avatars/765036bc276a1eb6ffb00e865eb2d40a.svg
Gongwei Chen
user
cgwfeel
66b426278a1be1e8acc2a71a
Dongmei Jiang
66b426278a1be1e8acc2a71b
Liqiang Nie
Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in
  Long-Horizon Tasks
Building a general-purpose agent is a long-standing vision in the field of
artificial intelligence. Existing agents have made remarkable progress in many
domains, yet they still struggle to complete long-horizon tasks in an open
world. We attribute this to the lack of necessary world knowledge and
multimodal experience that can guide agents through a variety of long-horizon
tasks. In this paper, we propose a Hybrid Multimodal Memory module to address
the above challenges. It 1) transforms knowledge into Hierarchical Directed
Knowledge Graph that allows agents to explicitly represent and learn world
knowledge, and 2) summarises historical information into Abstracted Multimodal
Experience Pool that provide agents with rich references for in-context
learning. On top of the Hybrid Multimodal Memory module, a multimodal agent,
Optimus-1, is constructed with dedicated Knowledge-guided Planner and
Experience-Driven Reflector, contributing to a better planning and reflection
in the face of long-horizon tasks in Minecraft. Extensive experimental results
show that Optimus-1 significantly outperforms all existing agents on
challenging long-horizon task benchmarks, and exhibits near human-level
performance on many tasks. In addition, we introduce various Multimodal Large
Language Models (MLLMs) as the backbone of Optimus-1. Experimental results show
that Optimus-1 exhibits strong generalization with the help of the Hybrid
Multimodal Memory module, outperforming the GPT-4V baseline on many tasks.
A multimodal agent, Optimus-1, equipped with a Hybrid Multimodal Memory module, significantly outperforms existing agents in long-horizon tasks by leveraging hierarchical knowledge graphs and abstracted multimodal experience pools.
Hierarchical Directed Knowledge Graph Abstracted Multimodal Experience Pool Hybrid Multimodal Memory Knowledge-guided Planner Experience-Driven Reflector Multimodal Large Language Models MLLMs