title: APE: Faster and Longer Context-Augmented Generation via Adaptive
    Parallel Encoding
summary: Context-augmented generation (CAG) techniques, including RAG and ICL, require
  the efficient combination of multiple contexts to generate responses to user
  queries. Directly inputting these contexts as a sequence introduces a
  considerable computational burden by re-encoding the combined selection of
  contexts for every request. To address this, we explore the promising potential
  of parallel encoding to independently pre-compute and cache each context's KV
  states. This approach enables the direct loading of cached states during
  inference while accommodating more contexts through position reuse across
  contexts. However, due to misalignments in attention distribution, directly
  applying parallel encoding results in a significant performance drop. To enable
  effective and efficient CAG, we propose Adaptive Parallel Encoding
  (APE), which brings shared prefix, attention temperature, and
  scaling factor to align the distribution of parallel encoding with sequential
  encoding. Results on RAG and ICL tasks demonstrate that APE can preserve 98%
  and 93% sequential encoding performance using the same inputs while
  outperforming parallel encoding by 3.6% and 7.9%, respectively. It also scales
  to many-shot CAG, effectively encoding hundreds of contexts in parallel.
  Efficiency evaluation shows that APE can achieve an end-to-end 4.5times
  speedup by reducing 28times prefilling time for a 128K-length context.
paper_authors._id: 67aac392385da1f07cc7fcbd
paper_authors.name: Xinyu Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T07:55:13.131Z
paper_authors._id: 64f58b970b24e548a85522bc
paper_authors.avatarUrl: /avatars/c8ca1294b5a1edd609694877e335b22f.svg
paper_authors.fullname: Xinyu Yang
paper_authors.type: user
paper_authors.user: Hanyuezhuohua
paper_authors._id: 67aac392385da1f07cc7fcbe
paper_authors.name: Tianqi Chen
paper_authors._id: 67aac392385da1f07cc7fcbf
paper_authors.name: Beidi Chen
paper_title: APE: Faster and Longer Context-Augmented Generation via Adaptive
    Parallel Encoding
paper_summary: Context-augmented generation (CAG) techniques, including RAG and ICL, require
  the efficient combination of multiple contexts to generate responses to user
  queries. Directly inputting these contexts as a sequence introduces a
  considerable computational burden by re-encoding the combined selection of
  contexts for every request. To address this, we explore the promising potential
  of parallel encoding to independently pre-compute and cache each context's KV
  states. This approach enables the direct loading of cached states during
  inference while accommodating more contexts through position reuse across
  contexts. However, due to misalignments in attention distribution, directly
  applying parallel encoding results in a significant performance drop. To enable
  effective and efficient CAG, we propose Adaptive Parallel Encoding
  (APE), which brings shared prefix, attention temperature, and
  scaling factor to align the distribution of parallel encoding with sequential
  encoding. Results on RAG and ICL tasks demonstrate that APE can preserve 98%
  and 93% sequential encoding performance using the same inputs while
  outperforming parallel encoding by 3.6% and 7.9%, respectively. It also scales
  to many-shot CAG, effectively encoding hundreds of contexts in parallel.
  Efficiency evaluation shows that APE can achieve an end-to-end 4.5times
  speedup by reducing 28times prefilling time for a 128K-length context.
paper_ai_summary: Adaptive Parallel Encoding (APE) improves context-augmented generation by aligning parallel encoding with sequential encoding, preserving performance while significantly increasing efficiency and scalability.
paper_ai_keywords: Context-augmented generation RAG ICL parallel encoding KV states position reuse attention distribution shared prefix attention temperature scaling factor many-shot CAG prefilling time