title: Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in
    Long-Horizon Tasks
summary: Building a general-purpose agent is a long-standing vision in the field of
  artificial intelligence. Existing agents have made remarkable progress in many
  domains, yet they still struggle to complete long-horizon tasks in an open
  world. We attribute this to the lack of necessary world knowledge and
  multimodal experience that can guide agents through a variety of long-horizon
  tasks. In this paper, we propose a Hybrid Multimodal Memory module to address
  the above challenges. It 1) transforms knowledge into Hierarchical Directed
  Knowledge Graph that allows agents to explicitly represent and learn world
  knowledge, and 2) summarises historical information into Abstracted Multimodal
  Experience Pool that provide agents with rich references for in-context
  learning. On top of the Hybrid Multimodal Memory module, a multimodal agent,
  Optimus-1, is constructed with dedicated Knowledge-guided Planner and
  Experience-Driven Reflector, contributing to a better planning and reflection
  in the face of long-horizon tasks in Minecraft. Extensive experimental results
  show that Optimus-1 significantly outperforms all existing agents on
  challenging long-horizon task benchmarks, and exhibits near human-level
  performance on many tasks. In addition, we introduce various Multimodal Large
  Language Models (MLLMs) as the backbone of Optimus-1. Experimental results show
  that Optimus-1 exhibits strong generalization with the help of the Hybrid
  Multimodal Memory module, outperforming the GPT-4V baseline on many tasks.
paper_authors._id: 66b426278a1be1e8acc2a716
paper_authors.name: Zaijing Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-08T10:13:09.178Z
paper_authors._id: 66b45fe75d0ac130d7d82764
paper_authors.avatarUrl: /avatars/09253f41f82c533b36199f82620cd075.svg
paper_authors.fullname: Zaijing Li
paper_authors.type: user
paper_authors.user: dawn0815
paper_authors._id: 66b426278a1be1e8acc2a717
paper_authors.name: Yuquan Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:13:23.374Z
paper_authors._id: 64b8e4760de7289de37450a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/q0q7LctHeuD-qtTZOtOP7.jpeg
paper_authors.fullname: Yuquan Xie
paper_authors.type: user
paper_authors.user: xieyuquan
paper_authors._id: 66b426278a1be1e8acc2a718
paper_authors.name: Rui Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:13:33.784Z
paper_authors._id: 64cf57956f107411da90ee42
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64cf57956f107411da90ee42/CQc3W9trxFDJiN2ZAcaxB.jpeg
paper_authors.fullname: Rui Shao
paper_authors.type: user
paper_authors.user: rshaojimmy
paper_authors._id: 66b426278a1be1e8acc2a719
paper_authors.name: Gongwei Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-08T10:52:29.117Z
paper_authors._id: 66b4a3575fa7debf39254e5c
paper_authors.avatarUrl: /avatars/765036bc276a1eb6ffb00e865eb2d40a.svg
paper_authors.fullname: Gongwei Chen
paper_authors.type: user
paper_authors.user: cgwfeel
paper_authors._id: 66b426278a1be1e8acc2a71a
paper_authors.name: Dongmei Jiang
paper_authors._id: 66b426278a1be1e8acc2a71b
paper_authors.name: Liqiang Nie
paper_title: Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in
    Long-Horizon Tasks
paper_summary: Building a general-purpose agent is a long-standing vision in the field of
  artificial intelligence. Existing agents have made remarkable progress in many
  domains, yet they still struggle to complete long-horizon tasks in an open
  world. We attribute this to the lack of necessary world knowledge and
  multimodal experience that can guide agents through a variety of long-horizon
  tasks. In this paper, we propose a Hybrid Multimodal Memory module to address
  the above challenges. It 1) transforms knowledge into Hierarchical Directed
  Knowledge Graph that allows agents to explicitly represent and learn world
  knowledge, and 2) summarises historical information into Abstracted Multimodal
  Experience Pool that provide agents with rich references for in-context
  learning. On top of the Hybrid Multimodal Memory module, a multimodal agent,
  Optimus-1, is constructed with dedicated Knowledge-guided Planner and
  Experience-Driven Reflector, contributing to a better planning and reflection
  in the face of long-horizon tasks in Minecraft. Extensive experimental results
  show that Optimus-1 significantly outperforms all existing agents on
  challenging long-horizon task benchmarks, and exhibits near human-level
  performance on many tasks. In addition, we introduce various Multimodal Large
  Language Models (MLLMs) as the backbone of Optimus-1. Experimental results show
  that Optimus-1 exhibits strong generalization with the help of the Hybrid
  Multimodal Memory module, outperforming the GPT-4V baseline on many tasks.
paper_ai_keywords: Hybrid Multimodal Memory module Hierarchical Directed Knowledge Graph Abstracted Multimodal Experience Pool Optimus-1 Knowledge-guided Planner Experience-Driven Reflector Multimodal Large Language Models (MLLMs) GPT-4V