title: MLLM as Retriever: Interactively Learning Multimodal Retrieval for
    Embodied Agents
summary: MLLM agents demonstrate potential for complex embodied tasks by retrieving
  multimodal task-relevant trajectory data. However, current retrieval methods
  primarily focus on surface-level similarities of textual or visual cues in
  trajectories, neglecting their effectiveness for the specific task at hand. To
  address this issue, we propose a novel method, MLLM as ReTriever (MART), which
  enhances the performance of embodied agents by utilizing interaction data to
  fine-tune an MLLM retriever based on preference learning, such that the
  retriever fully considers the effectiveness of trajectories and prioritize them
  for unseen tasks. We also introduce Trajectory Abstraction, a mechanism that
  leverages MLLMs' summarization capabilities to represent trajectories with
  fewer tokens while preserving key information, enabling agents to better
  comprehend milestones in the trajectory. Experimental results across various
  environments demonstrate our method significantly improves task success rates
  in unseen scenes compared to baseline methods. This work presents a new
  paradigm for multimodal retrieval in embodied agents, by fine-tuning a
  general-purpose MLLM as the retriever to assess trajectory effectiveness. All
  benchmark task sets and simulator code modifications for action and observation
  spaces will be released.
paper_authors._id: 6708926bd08850e483b2e54b
paper_authors.name: Junpeng Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:10:59.508Z
paper_authors._id: 6708e5b6761c066c92fe04d4
paper_authors.avatarUrl: /avatars/86b1491ccdb2d050a53b7cbac85e2210.svg
paper_authors.fullname: yjp
paper_authors.type: user
paper_authors.user: yuejunpengpku
paper_authors._id: 6708926bd08850e483b2e54c
paper_authors.name: Xinru Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T14:58:21.694Z
paper_authors._id: 653238fdcd5377e9adee0c41
paper_authors.avatarUrl: /avatars/78aea70cde6ab0050c7e18b5e148075c.svg
paper_authors.fullname: Xinrun Xu
paper_authors.type: user
paper_authors.user: SherryXu
paper_authors._id: 6708926bd08850e483b2e54d
paper_authors.name: Börje F. Karlsson
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:43:38.085Z
paper_authors._id: 61e52be53d6dbb1da842316a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e52be53d6dbb1da842316a/gx0WGPcOCClXPymoKglc4.jpeg
paper_authors.fullname: Börje Karlsson
paper_authors.type: user
paper_authors.user: tellarin
paper_authors._id: 6708926bd08850e483b2e54e
paper_authors.name: Zongqing Lu
paper_title: MLLM as Retriever: Interactively Learning Multimodal Retrieval for
    Embodied Agents
paper_summary: MLLM agents demonstrate potential for complex embodied tasks by retrieving
  multimodal task-relevant trajectory data. However, current retrieval methods
  primarily focus on surface-level similarities of textual or visual cues in
  trajectories, neglecting their effectiveness for the specific task at hand. To
  address this issue, we propose a novel method, MLLM as ReTriever (MART), which
  enhances the performance of embodied agents by utilizing interaction data to
  fine-tune an MLLM retriever based on preference learning, such that the
  retriever fully considers the effectiveness of trajectories and prioritize them
  for unseen tasks. We also introduce Trajectory Abstraction, a mechanism that
  leverages MLLMs' summarization capabilities to represent trajectories with
  fewer tokens while preserving key information, enabling agents to better
  comprehend milestones in the trajectory. Experimental results across various
  environments demonstrate our method significantly improves task success rates
  in unseen scenes compared to baseline methods. This work presents a new
  paradigm for multimodal retrieval in embodied agents, by fine-tuning a
  general-purpose MLLM as the retriever to assess trajectory effectiveness. All
  benchmark task sets and simulator code modifications for action and observation
  spaces will be released.
paper_ai_summary: The proposed MART method enhances embodied agents by fine-tuning an MLLM retriever with preference learning to prioritize task-effective trajectories, improving success rates in unseen environments.
paper_ai_keywords: MLLM Retriever MLLM as ReTriever (MART) preference learning Trajectory Abstraction multimodal retrieval task-effective trajectories