title: Evaluating Very Long-Term Conversational Memory of LLM Agents
summary: Existing works on long-term open-domain dialogues focus on evaluating model
  responses within contexts spanning no more than five chat sessions. Despite
  advancements in long-context large language models (LLMs) and retrieval
  augmented generation (RAG) techniques, their efficacy in very long-term
  dialogues remains unexplored. To address this research gap, we introduce a
  machine-human pipeline to generate high-quality, very long-term dialogues by
  leveraging LLM-based agent architectures and grounding their dialogues on
  personas and temporal event graphs. Moreover, we equip each agent with the
  capability of sharing and reacting to images. The generated conversations are
  verified and edited by human annotators for long-range consistency and
  grounding to the event graphs. Using this pipeline, we collect LoCoMo, a
  dataset of very long-term conversations, each encompassing 300 turns and 9K
  tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a
  comprehensive evaluation benchmark to measure long-term memory in models,
  encompassing question answering, event summarization, and multi-modal dialogue
  generation tasks. Our experimental results indicate that LLMs exhibit
  challenges in understanding lengthy conversations and comprehending long-range
  temporal and causal dynamics within dialogues. Employing strategies like
  long-context LLMs or RAG can offer improvements but these models still
  substantially lag behind human performance.
paper_authors._id: 65ded42cf4cf5235192c2669
paper_authors.name: Adyasha Maharana
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:48:40.213Z
paper_authors._id: 631b5e5c824f2502e35161e7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663166686696-631b5e5c824f2502e35161e7.jpeg
paper_authors.fullname: Adyasha Maharana
paper_authors.type: user
paper_authors.user: adymaharana
paper_authors._id: 65ded42cf4cf5235192c266a
paper_authors.name: Dong-Ho Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:48:59.493Z
paper_authors._id: 6142ec5a7215c6d505bafd4e
paper_authors.avatarUrl: /avatars/ae0387b672435c5a4cf16ff6764ce597.svg
paper_authors.fullname: Dong-Ho Lee
paper_authors.type: user
paper_authors.user: danny911kr
paper_authors._id: 65ded42cf4cf5235192c266b
paper_authors.name: Sergey Tulyakov
paper_authors._id: 65ded42cf4cf5235192c266c
paper_authors.name: Mohit Bansal
paper_authors._id: 65ded42cf4cf5235192c266d
paper_authors.name: Francesco Barbieri
paper_authors._id: 65ded42cf4cf5235192c266e
paper_authors.name: Yuwei Fang
paper_title: Evaluating Very Long-Term Conversational Memory of LLM Agents
paper_summary: Existing works on long-term open-domain dialogues focus on evaluating model
  responses within contexts spanning no more than five chat sessions. Despite
  advancements in long-context large language models (LLMs) and retrieval
  augmented generation (RAG) techniques, their efficacy in very long-term
  dialogues remains unexplored. To address this research gap, we introduce a
  machine-human pipeline to generate high-quality, very long-term dialogues by
  leveraging LLM-based agent architectures and grounding their dialogues on
  personas and temporal event graphs. Moreover, we equip each agent with the
  capability of sharing and reacting to images. The generated conversations are
  verified and edited by human annotators for long-range consistency and
  grounding to the event graphs. Using this pipeline, we collect LoCoMo, a
  dataset of very long-term conversations, each encompassing 300 turns and 9K
  tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a
  comprehensive evaluation benchmark to measure long-term memory in models,
  encompassing question answering, event summarization, and multi-modal dialogue
  generation tasks. Our experimental results indicate that LLMs exhibit
  challenges in understanding lengthy conversations and comprehending long-range
  temporal and causal dynamics within dialogues. Employing strategies like
  long-context LLMs or RAG can offer improvements but these models still
  substantially lag behind human performance.
paper_ai_keywords: long-term open-domain dialogues large language models (LLMs) retrieval augmented generation (RAG) machine-human pipeline long-context LLMs personas temporal event graphs multi-modal dialogue generation tasks question answering event summarization long-range memory long-term memory temporal and causal dynamics