title: LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive
    Memory
summary: Recent large language model (LLM)-driven chat assistant systems have
  integrated memory components to track user-assistant chat histories, enabling
  more accurate and personalized responses. However, their long-term memory
  capabilities in sustained interactions remain underexplored. This paper
  introduces LongMemEval, a comprehensive benchmark designed to evaluate five
  core long-term memory abilities of chat assistants: information extraction,
  multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
  With 500 meticulously curated questions embedded within freely scalable
  user-assistant chat histories, LongMemEval presents a significant challenge to
  existing long-term memory systems, with commercial chat assistants and
  long-context LLMs showing 30% accuracy drop on memorizing information across
  sustained interactions. We then present a unified framework that breaks down
  the long-term memory design into four design choices across the indexing,
  retrieval, and reading stages. Built upon key experimental insights, we propose
  several memory designs including session decomposition for optimizing value
  granularity, fact-augmented key expansion for enhancing the index structure,
  and time-aware query expansion for refining the search scope. Experiment
  results show that these optimizations greatly improve both memory recall and
  downstream question answering on LongMemEval. Overall, our study provides
  valuable resources and guidance for advancing the long-term memory capabilities
  of LLM-based chat assistants, paving the way toward more personalized and
  reliable conversational AI.
paper_authors._id: 670defb480363ae01368542c
paper_authors.name: Di Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T08:07:50.570Z
paper_authors._id: 639bf367445b133a4e97ef9c
paper_authors.avatarUrl: /avatars/51b59f4616a01796e07c05c9aa5286f8.svg
paper_authors.fullname: Di Wu
paper_authors.type: user
paper_authors.user: xiaowu0162
paper_authors._id: 670defb480363ae01368542d
paper_authors.name: Hongwei Wang
paper_authors._id: 670defb480363ae01368542e
paper_authors.name: Wenhao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T16:41:35.824Z
paper_authors._id: 5feab3a28a3201f8e554c969
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
paper_authors.fullname: Wenhao Yu
paper_authors.type: user
paper_authors.user: wyu1
paper_authors._id: 670defb480363ae01368542f
paper_authors.name: Yuwei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T21:17:57.174Z
paper_authors._id: 663187c1a2354b0f50ab10a0
paper_authors.avatarUrl: /avatars/bef0fd9d2afa6a4990bc32bd55cbe163.svg
paper_authors.fullname: Yuwei Zhang
paper_authors.type: user
paper_authors.user: YWZBrandon
paper_authors._id: 670defb480363ae013685430
paper_authors.name: Kai-Wei Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T16:42:29.388Z
paper_authors._id: 60b7b9d71b90c5d07c23fbd0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
paper_authors.fullname: Kai-Wei Chang
paper_authors.type: user
paper_authors.user: kaiweichang
paper_authors._id: 670defb480363ae013685431
paper_authors.name: Dong Yu
paper_title: LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive
    Memory
paper_summary: Recent large language model (LLM)-driven chat assistant systems have
  integrated memory components to track user-assistant chat histories, enabling
  more accurate and personalized responses. However, their long-term memory
  capabilities in sustained interactions remain underexplored. This paper
  introduces LongMemEval, a comprehensive benchmark designed to evaluate five
  core long-term memory abilities of chat assistants: information extraction,
  multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
  With 500 meticulously curated questions embedded within freely scalable
  user-assistant chat histories, LongMemEval presents a significant challenge to
  existing long-term memory systems, with commercial chat assistants and
  long-context LLMs showing 30% accuracy drop on memorizing information across
  sustained interactions. We then present a unified framework that breaks down
  the long-term memory design into four design choices across the indexing,
  retrieval, and reading stages. Built upon key experimental insights, we propose
  several memory designs including session decomposition for optimizing value
  granularity, fact-augmented key expansion for enhancing the index structure,
  and time-aware query expansion for refining the search scope. Experiment
  results show that these optimizations greatly improve both memory recall and
  downstream question answering on LongMemEval. Overall, our study provides
  valuable resources and guidance for advancing the long-term memory capabilities
  of LLM-based chat assistants, paving the way toward more personalized and
  reliable conversational AI.
paper_ai_keywords: information extraction multi-core reasoning contemporary reasoning knowledge updates abstention long-term memory long-context chat histories memory recall question answering long-term memory designs sessions decomposition fact-augmented key expansion time-aware query expansion