Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
Recent benchmarks for Large Language Model (LLM) agents primarily focus on
evaluating reasoning, planning, and execution capabilities, while another
critical component-memory, encompassing how agents memorize, update, and
retrieve long-term information-is under-evaluated due to the lack of
benchmarks. We term agents with memory mechanisms as memory agents. In this
paper, we identify four core competencies essential for memory agents: accurate
retrieval, test-time learning, long-range understanding, and conflict
resolution. Existing datasets either rely on limited context lengths or are
tailored for static, long-context settings like book-based QA, which do not
reflect the interactive, multi-turn nature of memory agents that incrementally
accumulate information. Furthermore, no existing benchmarks cover all four
competencies. Therefore, we introduce MemoryAgentBench, a new benchmark
specifically designed for memory agents. Our benchmark combines reformulated
existing datasets with newly constructed ones, covering the above four memory
competencies, providing a systematic and challenging testbed for assessing
memory quality. We evaluate a diverse set of memory agents, ranging from simple
context-based and retrieval-augmented generation (RAG) systems to advanced
agents with external memory modules and tool integration. Empirical results
reveal that current methods fall short of mastering all four competencies,
underscoring the need for further research into comprehensive memory mechanisms
for LLM agents.
686d7fa3cb5725779c60b262
Yuanzhe Hu
claimed_verified
2025-07-09T08:51:02.714Z
686647f3e2cca4bc45818704
/avatars/1efeff78892223b6e5592d4ef994f7c3.svg
YUANZHE HU
user
ai-hyz
686d7fa3cb5725779c60b263
Yu Wang
claimed_verified
2025-07-11T08:03:57.705Z
63234809155b0e2c44f354d6
/avatars/60d38f8f0e12363f3f5e0388e635d7b6.svg
Yu Wang
user
YuWangX
686d7fa3cb5725779c60b264
Julian McAuley
Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
Recent benchmarks for Large Language Model (LLM) agents primarily focus on
evaluating reasoning, planning, and execution capabilities, while another
critical component-memory, encompassing how agents memorize, update, and
retrieve long-term information-is under-evaluated due to the lack of
benchmarks. We term agents with memory mechanisms as memory agents. In this
paper, we identify four core competencies essential for memory agents: accurate
retrieval, test-time learning, long-range understanding, and conflict
resolution. Existing datasets either rely on limited context lengths or are
tailored for static, long-context settings like book-based QA, which do not
reflect the interactive, multi-turn nature of memory agents that incrementally
accumulate information. Furthermore, no existing benchmarks cover all four
competencies. Therefore, we introduce MemoryAgentBench, a new benchmark
specifically designed for memory agents. Our benchmark combines reformulated
existing datasets with newly constructed ones, covering the above four memory
competencies, providing a systematic and challenging testbed for assessing
memory quality. We evaluate a diverse set of memory agents, ranging from simple
context-based and retrieval-augmented generation (RAG) systems to advanced
agents with external memory modules and tool integration. Empirical results
reveal that current methods fall short of mastering all four competencies,
underscoring the need for further research into comprehensive memory mechanisms
for LLM agents.
MemoryAgentBench is a new benchmark designed to evaluate four core competencies of memory agents in Large Language Models, highlighting the need for improved memory mechanisms.
memory agents accurate retrieval test-time learning long-range understanding conflict resolution context-based retrieval-augmented generation external memory modules tool integration