title: QwenLong-CPRS: Towards infty-LLMs with Dynamic Context Optimization
summary: This technical report presents QwenLong-CPRS, a context compression framework
  designed for explicit long-context optimization, addressing prohibitive
  computation overhead during the prefill stage and the "lost in the middle"
  performance degradation of large language models (LLMs) during long sequence
  processing. Implemented through a novel dynamic context optimization mechanism,
  QwenLong-CPRS enables multi-granularity context compression guided by natural
  language instructions, achieving both efficiency gains and improved
  performance.
    Evolved from the Qwen architecture series, QwenLong-CPRS introduces four key
  innovations: (1) Natural language-guided dynamic optimization, (2)
  Bidirectional reasoning layers for enhanced boundary awareness, (3) Token
  critic mechanisms with language modeling heads, and (4) Window-parallel
  inference.
    Comprehensive evaluations across five benchmarks (4K-2M word contexts)
  demonstrate QwenLong-CPRS's threefold effectiveness: (1) Consistent superiority
  over other context management methods like RAG and sparse attention in both
  accuracy and efficiency. (2) Architecture-agnostic integration with all
  flagship LLMs, including GPT-4o, Gemini2.0-pro, Claude3.7-sonnet, DeepSeek-v3,
  and Qwen2.5-max, achieves 21.59times context compression alongside
  19.15-point average performance gains; (3) Deployed with Qwen2.5-32B-Instruct,
  QwenLong-CPRS surpasses leading proprietary LLMs by 4.85 and 10.88 points on
  Ruler-128K and InfiniteBench, establishing new SOTA performance.
paper_authors._id: 6833ea049f968fc5c6b64486
paper_authors.name: Weizhou Shen
paper_authors._id: 6833ea049f968fc5c6b64487
paper_authors.name: Chenliang Li
paper_authors._id: 6833ea049f968fc5c6b64488
paper_authors.name: Fanqi Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:09:25.991Z
paper_authors._id: 62ecbffd99112e99c5f7fded
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62ecbffd99112e99c5f7fded/U6iXAJbpm2vaC5qksEPiH.png
paper_authors.fullname: Fanqi Wan
paper_authors.type: user
paper_authors.user: Wanfq
paper_authors._id: 6833ea049f968fc5c6b64489
paper_authors.name: Shengyi Liao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T12:43:57.415Z
paper_authors._id: 64a69c1a9704e9888c1da827
paper_authors.avatarUrl: /avatars/096aefab0ca99af76097cdfc5e752402.svg
paper_authors.fullname: LiaoShengyi
paper_authors.type: user
paper_authors.user: LsyLsyLsyyy
paper_authors._id: 6833ea049f968fc5c6b6448a
paper_authors.name: Shaopeng Lai
paper_authors._id: 6833ea049f968fc5c6b6448b
paper_authors.name: Bo Zhang
paper_authors._id: 6833ea049f968fc5c6b6448c
paper_authors.name: Yingcheng Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:54:57.679Z
paper_authors._id: 64a3897a34612d376415545c
paper_authors.avatarUrl: /avatars/3d69ce7f59783e51fd8a97830333ead6.svg
paper_authors.fullname: shiyingcheng
paper_authors.type: user
paper_authors.user: shiyingcheng
paper_authors._id: 6833ea049f968fc5c6b6448d
paper_authors.name: Yuning Wu
paper_authors._id: 6833ea049f968fc5c6b6448e
paper_authors.name: Gang Fu
paper_authors._id: 6833ea049f968fc5c6b6448f
paper_authors.name: Zhansheng Li
paper_authors._id: 6833ea049f968fc5c6b64490
paper_authors.name: Bin Yang
paper_authors._id: 6833ea049f968fc5c6b64491
paper_authors.name: Ji Zhang
paper_authors._id: 6833ea049f968fc5c6b64492
paper_authors.name: Fei Huang
paper_authors._id: 6833ea049f968fc5c6b64493
paper_authors.name: Jingren Zhou
paper_authors._id: 6833ea049f968fc5c6b64494
paper_authors.name: Ming Yan
paper_title: QwenLong-CPRS: Towards infty-LLMs with Dynamic Context Optimization
paper_summary: This technical report presents QwenLong-CPRS, a context compression framework
  designed for explicit long-context optimization, addressing prohibitive
  computation overhead during the prefill stage and the "lost in the middle"
  performance degradation of large language models (LLMs) during long sequence
  processing. Implemented through a novel dynamic context optimization mechanism,
  QwenLong-CPRS enables multi-granularity context compression guided by natural
  language instructions, achieving both efficiency gains and improved
  performance.
    Evolved from the Qwen architecture series, QwenLong-CPRS introduces four key
  innovations: (1) Natural language-guided dynamic optimization, (2)
  Bidirectional reasoning layers for enhanced boundary awareness, (3) Token
  critic mechanisms with language modeling heads, and (4) Window-parallel
  inference.
    Comprehensive evaluations across five benchmarks (4K-2M word contexts)
  demonstrate QwenLong-CPRS's threefold effectiveness: (1) Consistent superiority
  over other context management methods like RAG and sparse attention in both
  accuracy and efficiency. (2) Architecture-agnostic integration with all
  flagship LLMs, including GPT-4o, Gemini2.0-pro, Claude3.7-sonnet, DeepSeek-v3,
  and Qwen2.5-max, achieves 21.59times context compression alongside
  19.15-point average performance gains; (3) Deployed with Qwen2.5-32B-Instruct,
  QwenLong-CPRS surpasses leading proprietary LLMs by 4.85 and 10.88 points on
  Ruler-128K and InfiniteBench, establishing new SOTA performance.
paper_ai_summary: QwenLong-CPRS enhances large language models with multi-granularity context compression, dynamic optimization guided by natural language, and efficient bidirectional reasoning and parallel inference, achieving superior performance and context management.
paper_ai_keywords: context compression dynamic context optimization bidirectional reasoning layers token critic mechanisms window-parallel inference Qwen RAG sparse attention large language models SOTA performance