MLLM as Retriever: Interactively Learning Multimodal Retrieval for
  Embodied Agents
MLLM agents demonstrate potential for complex embodied tasks by retrieving
multimodal task-relevant trajectory data. However, current retrieval methods
primarily focus on surface-level similarities of textual or visual cues in
trajectories, neglecting their effectiveness for the specific task at hand. To
address this issue, we propose a novel method, MLLM as ReTriever (MART), which
enhances the performance of embodied agents by utilizing interaction data to
fine-tune an MLLM retriever based on preference learning, such that the
retriever fully considers the effectiveness of trajectories and prioritize them
for unseen tasks. We also introduce Trajectory Abstraction, a mechanism that
leverages MLLMs' summarization capabilities to represent trajectories with
fewer tokens while preserving key information, enabling agents to better
comprehend milestones in the trajectory. Experimental results across various
environments demonstrate our method significantly improves task success rates
in unseen scenes compared to baseline methods. This work presents a new
paradigm for multimodal retrieval in embodied agents, by fine-tuning a
general-purpose MLLM as the retriever to assess trajectory effectiveness. All
benchmark task sets and simulator code modifications for action and observation
spaces will be released.
6708926bd08850e483b2e54b
Junpeng Yue
admin_assigned
2024-10-11T10:10:59.508Z
6708e5b6761c066c92fe04d4
/avatars/86b1491ccdb2d050a53b7cbac85e2210.svg
yjp
user
yuejunpengpku
6708926bd08850e483b2e54c
Xinru Xu
claimed_verified
2025-03-18T14:58:21.694Z
653238fdcd5377e9adee0c41
/avatars/78aea70cde6ab0050c7e18b5e148075c.svg
Xinrun Xu
user
SherryXu
6708926bd08850e483b2e54d
Börje F. Karlsson
claimed_verified
2024-10-11T07:43:38.085Z
61e52be53d6dbb1da842316a
https://cdn-avatars.huggingface.co/v1/production/uploads/61e52be53d6dbb1da842316a/gx0WGPcOCClXPymoKglc4.jpeg
Börje Karlsson
user
tellarin
6708926bd08850e483b2e54e
Zongqing Lu
MLLM as Retriever: Interactively Learning Multimodal Retrieval for
  Embodied Agents
MLLM agents demonstrate potential for complex embodied tasks by retrieving
multimodal task-relevant trajectory data. However, current retrieval methods
primarily focus on surface-level similarities of textual or visual cues in
trajectories, neglecting their effectiveness for the specific task at hand. To
address this issue, we propose a novel method, MLLM as ReTriever (MART), which
enhances the performance of embodied agents by utilizing interaction data to
fine-tune an MLLM retriever based on preference learning, such that the
retriever fully considers the effectiveness of trajectories and prioritize them
for unseen tasks. We also introduce Trajectory Abstraction, a mechanism that
leverages MLLMs' summarization capabilities to represent trajectories with
fewer tokens while preserving key information, enabling agents to better
comprehend milestones in the trajectory. Experimental results across various
environments demonstrate our method significantly improves task success rates
in unseen scenes compared to baseline methods. This work presents a new
paradigm for multimodal retrieval in embodied agents, by fine-tuning a
general-purpose MLLM as the retriever to assess trajectory effectiveness. All
benchmark task sets and simulator code modifications for action and observation
spaces will be released.
The proposed MART method enhances embodied agents by fine-tuning an MLLM retriever with preference learning to prioritize task-effective trajectories, improving success rates in unseen environments.
MLLM Retriever MLLM as ReTriever (MART) preference learning Trajectory Abstraction multimodal retrieval task-effective trajectories