From RAG to Memory: Non-Parametric Continual Learning for Large Language
  Models
Our ability to continuously acquire, organize, and leverage knowledge is a
key feature of human intelligence that AI systems must approximate to unlock
their full potential. Given the challenges in continual learning with large
language models (LLMs), retrieval-augmented generation (RAG) has become the
dominant way to introduce new information. However, its reliance on vector
retrieval hinders its ability to mimic the dynamic and interconnected nature of
human long-term memory. Recent RAG approaches augment vector embeddings with
various structures like knowledge graphs to address some of these gaps, namely
sense-making and associativity. However, their performance on more basic
factual memory tasks drops considerably below standard RAG. We address this
unintended deterioration and propose HippoRAG 2, a framework that outperforms
standard RAG comprehensively on factual, sense-making, and associative memory
tasks. HippoRAG 2 builds upon the Personalized PageRank algorithm used in
HippoRAG and enhances it with deeper passage integration and more effective
online use of an LLM. This combination pushes this RAG system closer to the
effectiveness of human long-term memory, achieving a 7% improvement in
associative memory tasks over the state-of-the-art embedding model while also
exhibiting superior factual knowledge and sense-making memory capabilities.
This work paves the way for non-parametric continual learning for LLMs. Our
code and data will be released at https://github.com/OSU-NLP-Group/HippoRAG.
67b878b8f17ca6989fd21e92
Bernal Jiménez Gutiérrez
67b878b8f17ca6989fd21e93
Yiheng Shu
admin_assigned
2025-02-21T15:15:54.523Z
60a4ebfbaa9320dbbe69e37c
https://cdn-avatars.huggingface.co/v1/production/uploads/60a4ebfbaa9320dbbe69e37c/QLaEohXCWaUy8YX3wKQ_w.jpeg
Yiheng Shu
user
yhshu
67b878b8f17ca6989fd21e94
Weijian Qi
claimed_verified
2025-02-24T09:21:41.970Z
67169e3fd720d7d51e36a67e
/avatars/91113e5520009a4ed709bc62b96f2150.svg
WeijianQi
user
WeijianQi1999
67b878b8f17ca6989fd21e95
Sizhe Zhou
admin_assigned
2025-02-21T15:16:08.161Z
65030fc90a57e8f2b26bcaa3
/avatars/8c25a4a9735f268ee8541f3e2017d92c.svg
Sizhe Zhou
user
KevinSRR
67b878b8f17ca6989fd21e96
Yu Su
From RAG to Memory: Non-Parametric Continual Learning for Large Language
  Models
Our ability to continuously acquire, organize, and leverage knowledge is a
key feature of human intelligence that AI systems must approximate to unlock
their full potential. Given the challenges in continual learning with large
language models (LLMs), retrieval-augmented generation (RAG) has become the
dominant way to introduce new information. However, its reliance on vector
retrieval hinders its ability to mimic the dynamic and interconnected nature of
human long-term memory. Recent RAG approaches augment vector embeddings with
various structures like knowledge graphs to address some of these gaps, namely
sense-making and associativity. However, their performance on more basic
factual memory tasks drops considerably below standard RAG. We address this
unintended deterioration and propose HippoRAG 2, a framework that outperforms
standard RAG comprehensively on factual, sense-making, and associative memory
tasks. HippoRAG 2 builds upon the Personalized PageRank algorithm used in
HippoRAG and enhances it with deeper passage integration and more effective
online use of an LLM. This combination pushes this RAG system closer to the
effectiveness of human long-term memory, achieving a 7% improvement in
associative memory tasks over the state-of-the-art embedding model while also
exhibiting superior factual knowledge and sense-making memory capabilities.
This work paves the way for non-parametric continual learning for LLMs. Our
code and data will be released at https://github.com/OSU-NLP-Group/HippoRAG.
HippoRAG 2 enhances RAG by integrating passage information and leveraging a more effective LLM, improving performances across factual, sense-making, and associative memory tasks.
retrieval-augmented generation (RAG) vector retrieval knowledge graphs Personalized PageRank passage integration non-parametric continual learning large language models (LLMs)