Evaluating Very Long-Term Conversational Memory of LLM Agents
Existing works on long-term open-domain dialogues focus on evaluating model
responses within contexts spanning no more than five chat sessions. Despite
advancements in long-context large language models (LLMs) and retrieval
augmented generation (RAG) techniques, their efficacy in very long-term
dialogues remains unexplored. To address this research gap, we introduce a
machine-human pipeline to generate high-quality, very long-term dialogues by
leveraging LLM-based agent architectures and grounding their dialogues on
personas and temporal event graphs. Moreover, we equip each agent with the
capability of sharing and reacting to images. The generated conversations are
verified and edited by human annotators for long-range consistency and
grounding to the event graphs. Using this pipeline, we collect LoCoMo, a
dataset of very long-term conversations, each encompassing 300 turns and 9K
tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a
comprehensive evaluation benchmark to measure long-term memory in models,
encompassing question answering, event summarization, and multi-modal dialogue
generation tasks. Our experimental results indicate that LLMs exhibit
challenges in understanding lengthy conversations and comprehending long-range
temporal and causal dynamics within dialogues. Employing strategies like
long-context LLMs or RAG can offer improvements but these models still
substantially lag behind human performance.
65ded42cf4cf5235192c2669
Adyasha Maharana
admin_assigned
2024-02-28T12:48:40.213Z
631b5e5c824f2502e35161e7
https://cdn-avatars.huggingface.co/v1/production/uploads/1663166686696-631b5e5c824f2502e35161e7.jpeg
Adyasha Maharana
user
adymaharana
65ded42cf4cf5235192c266a
Dong-Ho Lee
admin_assigned
2024-02-28T12:48:59.493Z
6142ec5a7215c6d505bafd4e
/avatars/ae0387b672435c5a4cf16ff6764ce597.svg
Dong-Ho Lee
user
danny911kr
65ded42cf4cf5235192c266b
Sergey Tulyakov
65ded42cf4cf5235192c266c
Mohit Bansal
65ded42cf4cf5235192c266d
Francesco Barbieri
65ded42cf4cf5235192c266e
Yuwei Fang
Evaluating Very Long-Term Conversational Memory of LLM Agents
Existing works on long-term open-domain dialogues focus on evaluating model
responses within contexts spanning no more than five chat sessions. Despite
advancements in long-context large language models (LLMs) and retrieval
augmented generation (RAG) techniques, their efficacy in very long-term
dialogues remains unexplored. To address this research gap, we introduce a
machine-human pipeline to generate high-quality, very long-term dialogues by
leveraging LLM-based agent architectures and grounding their dialogues on
personas and temporal event graphs. Moreover, we equip each agent with the
capability of sharing and reacting to images. The generated conversations are
verified and edited by human annotators for long-range consistency and
grounding to the event graphs. Using this pipeline, we collect LoCoMo, a
dataset of very long-term conversations, each encompassing 300 turns and 9K
tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a
comprehensive evaluation benchmark to measure long-term memory in models,
encompassing question answering, event summarization, and multi-modal dialogue
generation tasks. Our experimental results indicate that LLMs exhibit
challenges in understanding lengthy conversations and comprehending long-range
temporal and causal dynamics within dialogues. Employing strategies like
long-context LLMs or RAG can offer improvements but these models still
substantially lag behind human performance.
A pipeline combining LLMs and human annotation generates long-term dialogues to evaluate model performance in understanding lengthy conversations and temporal dynamics.
long-term open-domain dialogues long-context large language models retrieval augmented generation personas temporal event graphs multi-modal dialogue generation LoCoMo long-term memory event summarization