title: A Silver Bullet or a Compromise for Full Attention? A Comprehensive
    Study of Gist Token-based Context Compression
summary: In this work, we provide a thorough investigation of gist-based context
  compression methods to improve long-context processing in large language
  models. We focus on two key questions: (1) How well can these methods replace
  full attention models? and (2) What potential failure patterns arise due to
  compression? Through extensive experiments, we show that while gist-based
  compression can achieve near-lossless performance on tasks like
  retrieval-augmented generation and long-document QA, it faces challenges in
  tasks like synthetic recall. Furthermore, we identify three key failure
  patterns: lost by the boundary, lost if surprise, and lost along the way. To
  mitigate these issues, we propose two effective strategies: fine-grained
  autoencoding, which enhances the reconstruction of original token information,
  and segment-wise token importance estimation, which adjusts optimization based
  on token dependencies. Our work provides valuable insights into the
  understanding of gist token-based context compression and offers practical
  strategies for improving compression capabilities.
paper_authors._id: 676a2a3ebce62ec5a02c4a66
paper_authors.name: Chenlong Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-26T18:49:33.308Z
paper_authors._id: 654c99d6e82a71cb487c2ecd
paper_authors.avatarUrl: /avatars/480c883b587ed2e41e1e9661c844a938.svg
paper_authors.fullname: ChenlongDeng
paper_authors.type: user
paper_authors.user: ChenlongDeng
paper_authors._id: 676a2a3ebce62ec5a02c4a67
paper_authors.name: Zhisong Zhang
paper_authors._id: 676a2a3ebce62ec5a02c4a68
paper_authors.name: Kelong Mao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:35:30.975Z
paper_authors._id: 649a65605c74a2125c22bbc1
paper_authors.avatarUrl: /avatars/e7435d3aeeb59acc6f6f43b48d6982a0.svg
paper_authors.fullname: Mao
paper_authors.type: user
paper_authors.user: kyriemao
paper_authors._id: 676a2a3ebce62ec5a02c4a69
paper_authors.name: Shuaiyi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:35:28.792Z
paper_authors._id: 64d708e45e5f05485ce5fbd8
paper_authors.avatarUrl: /avatars/3c5389a5ffecbc05583004c627b21b6c.svg
paper_authors.fullname: Li Shuaiyi
paper_authors.type: user
paper_authors.user: Syon-Li
paper_authors._id: 676a2a3ebce62ec5a02c4a6a
paper_authors.name: Xinting Huang
paper_authors._id: 676a2a3ebce62ec5a02c4a6b
paper_authors.name: Dong Yu
paper_authors._id: 676a2a3ebce62ec5a02c4a6c
paper_authors.name: Zhicheng Dou
paper_title: A Silver Bullet or a Compromise for Full Attention? A Comprehensive
    Study of Gist Token-based Context Compression
paper_summary: In this work, we provide a thorough investigation of gist-based context
  compression methods to improve long-context processing in large language
  models. We focus on two key questions: (1) How well can these methods replace
  full attention models? and (2) What potential failure patterns arise due to
  compression? Through extensive experiments, we show that while gist-based
  compression can achieve near-lossless performance on tasks like
  retrieval-augmented generation and long-document QA, it faces challenges in
  tasks like synthetic recall. Furthermore, we identify three key failure
  patterns: lost by the boundary, lost if surprise, and lost along the way. To
  mitigate these issues, we propose two effective strategies: fine-grained
  autoencoding, which enhances the reconstruction of original token information,
  and segment-wise token importance estimation, which adjusts optimization based
  on token dependencies. Our work provides valuable insights into the
  understanding of gist token-based context compression and offers practical
  strategies for improving compression capabilities.
paper_ai_summary: Gist-based context compression in large language models offers near-lossless performance on some tasks but fails on synthetic recall, with identified failure patterns and proposed strategies for improvement.
paper_ai_keywords: gist-based context compression long-context processing large language models full attention models retrieval-augmented generation long-document QA synthetic recall fine-grained autoencoding segment-wise token importance estimation