title: Augmenting Language Models with Long-Term Memory
summary: Existing large language models (LLMs) can only afford fix-sized inputs due to
  the input length limit, preventing them from utilizing rich long-context
  information from past inputs. To address this, we propose a framework, Language
  Models Augmented with Long-Term Memory (LongMem), which enables LLMs to
  memorize long history. We design a novel decoupled network architecture with
  the original backbone LLM frozen as a memory encoder and an adaptive residual
  side-network as a memory retriever and reader. Such a decoupled memory design
  can easily cache and update long-term past contexts for memory retrieval
  without suffering from memory staleness. Enhanced with memory-augmented
  adaptation training, LongMem can thus memorize long past context and use
  long-term memory for language modeling. The proposed memory retrieval module
  can handle unlimited-length context in its memory bank to benefit various
  downstream tasks. Typically, LongMem can enlarge the long-form memory to 65k
  tokens and thus cache many-shot extra demonstration examples as long-form
  memory for in-context learning. Experiments show that our method outperforms
  strong long-context models on ChapterBreak, a challenging long-context modeling
  benchmark, and achieves remarkable improvements on memory-augmented in-context
  learning over LLMs. The results demonstrate that the proposed method is
  effective in helping language models to memorize and utilize long-form
  contents. Our code is open-sourced at https://aka.ms/LongMem.
paper_authors._id: 6487cbc5de085085be87a58c
paper_authors.name: Weizhi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T12:26:08.192Z
paper_authors._id: 63d34004b734eaa4d4faeccf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63d34004b734eaa4d4faeccf/aH7F-xn4PMQjHHcvL-0vs.jpeg
paper_authors.fullname: Weizhi Wang
paper_authors.type: user
paper_authors.user: weizhiwang
paper_authors._id: 6487cbc5de085085be87a58d
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-13T17:39:32.637Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 6487cbc5de085085be87a58e
paper_authors.name: Hao Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-18T06:45:12.199Z
paper_authors._id: 608c49299d7c9519b4adae25
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/608c49299d7c9519b4adae25/pWSW52_QU7VD9hhDUHbmq.jpeg
paper_authors.fullname: Hao Cheng
paper_authors.type: user
paper_authors.user: kelvinih
paper_authors._id: 6487cbc5de085085be87a58f
paper_authors.name: Xiaodong Liu
paper_authors._id: 6487cbc5de085085be87a590
paper_authors.name: Xifeng Yan
paper_authors._id: 6487cbc5de085085be87a591
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T12:27:21.583Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 6487cbc5de085085be87a592
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T12:26:50.909Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Augmenting Language Models with Long-Term Memory
paper_summary: Existing large language models (LLMs) can only afford fix-sized inputs due to
  the input length limit, preventing them from utilizing rich long-context
  information from past inputs. To address this, we propose a framework, Language
  Models Augmented with Long-Term Memory (LongMem), which enables LLMs to
  memorize long history. We design a novel decoupled network architecture with
  the original backbone LLM frozen as a memory encoder and an adaptive residual
  side-network as a memory retriever and reader. Such a decoupled memory design
  can easily cache and update long-term past contexts for memory retrieval
  without suffering from memory staleness. Enhanced with memory-augmented
  adaptation training, LongMem can thus memorize long past context and use
  long-term memory for language modeling. The proposed memory retrieval module
  can handle unlimited-length context in its memory bank to benefit various
  downstream tasks. Typically, LongMem can enlarge the long-form memory to 65k
  tokens and thus cache many-shot extra demonstration examples as long-form
  memory for in-context learning. Experiments show that our method outperforms
  strong long-context models on ChapterBreak, a challenging long-context modeling
  benchmark, and achieves remarkable improvements on memory-augmented in-context
  learning over LLMs. The results demonstrate that the proposed method is
  effective in helping language models to memorize and utilize long-form
  contents. Our code is open-sourced at https://aka.ms/LongMem.
paper_ai_summary: A framework called LongMem enables large language models to utilize long-term memory, overcoming input length limitations and improving performance on long-context tasks.
paper_ai_keywords: language models long-term memory memory encoder adaptive residual side-network memory retrieval memory staleness memory-augmented adaptation training long-form memory in-context learning ChapterBreak benchmark memory-augmented in-context learning