title: Shifting AI Efficiency From Model-Centric to Data-Centric Compression
summary: The rapid advancement of large language models (LLMs) and multi-modal LLMs
  (MLLMs) has historically relied on model-centric scaling through increasing
  parameter counts from millions to hundreds of billions to drive performance
  gains. However, as we approach hardware limits on model size, the dominant
  computational bottleneck has fundamentally shifted to the quadratic cost of
  self-attention over long token sequences, now driven by ultra-long text
  contexts, high-resolution images, and extended videos. In this position paper,
  we argue that the focus of research for efficient AI is shifting from
  model-centric compression to data-centric compression. We position token
  compression as the new frontier, which improves AI efficiency via reducing the
  number of tokens during model training or inference. Through comprehensive
  analysis, we first examine recent developments in long-context AI across
  various domains and establish a unified mathematical framework for existing
  model efficiency strategies, demonstrating why token compression represents a
  crucial paradigm shift in addressing long-context overhead. Subsequently, we
  systematically review the research landscape of token compression, analyzing
  its fundamental benefits and identifying its compelling advantages across
  diverse scenarios. Furthermore, we provide an in-depth analysis of current
  challenges in token compression research and outline promising future
  directions. Ultimately, our work aims to offer a fresh perspective on AI
  efficiency, synthesize existing research, and catalyze innovative developments
  to address the challenges that increasing context lengths pose to the AI
  community's advancement.
paper_authors._id: 68353258d005e45149d2d384
paper_authors.name: Xuyang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T08:23:05.932Z
paper_authors._id: 66a0caa1a7a6ed88ad1c0ddf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66a0caa1a7a6ed88ad1c0ddf/WoOP24-ruuHy4ryNhRp0D.jpeg
paper_authors.fullname: Xuyang Liu
paper_authors.type: user
paper_authors.user: xuyang-liu16
paper_authors._id: 68353258d005e45149d2d385
paper_authors.name: Zichen Wen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:50:45.710Z
paper_authors._id: 653b8c3e97a4d71d950e2f20
paper_authors.avatarUrl: /avatars/b68880022e14556d0be58c69615db3be.svg
paper_authors.fullname: Zichen Wen
paper_authors.type: user
paper_authors.user: zichenwen
paper_authors._id: 68353258d005e45149d2d386
paper_authors.name: Shaobo Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:50:41.853Z
paper_authors._id: 66968099c952e09a4cb29f78
paper_authors.avatarUrl: /avatars/bd3a361fe5315e26e9ae328071704eed.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Steven-Shaobo
paper_authors._id: 68353258d005e45149d2d387
paper_authors.name: Junjie Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:50:48.148Z
paper_authors._id: 652f8642338c761caf474169
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/mq5jjqqNaFxVboWGDEocJ.jpeg
paper_authors.fullname: Junjie Chen
paper_authors.type: user
paper_authors.user: coderchen01
paper_authors._id: 68353258d005e45149d2d388
paper_authors.name: Zhishan Tao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:50:43.758Z
paper_authors._id: 679f280ffb07d74f084520b6
paper_authors.avatarUrl: /avatars/b378000f68c7faf8d4fee8074dd2db5b.svg
paper_authors.fullname: Zhishan Tao
paper_authors.type: user
paper_authors.user: Pppeach33
paper_authors._id: 68353258d005e45149d2d389
paper_authors.name: Yubo Wang
paper_authors._id: 68353258d005e45149d2d38a
paper_authors.name: Xiangqi Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:21:21.961Z
paper_authors._id: 64abcbfde144ba0eb9bb8419
paper_authors.avatarUrl: /avatars/6ccea0e755bad384aaabd5c455bd962e.svg
paper_authors.fullname: Xiangqi Jin
paper_authors.type: user
paper_authors.user: Lueci4er
paper_authors._id: 68353258d005e45149d2d38b
paper_authors.name: Chang Zou
paper_authors._id: 68353258d005e45149d2d38c
paper_authors.name: Yiyu Wang
paper_authors._id: 68353258d005e45149d2d38d
paper_authors.name: Chenfei Liao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:00:32.765Z
paper_authors._id: 6806464ed918f6d2fee2bc8b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6806464ed918f6d2fee2bc8b/rgpG2oO0m6PT0KltCF_Wf.jpeg
paper_authors.fullname: Chenfei Liao
paper_authors.type: user
paper_authors.user: Chenfei-Liao
paper_authors._id: 68353258d005e45149d2d38e
paper_authors.name: Xu Zheng
paper_authors._id: 68353258d005e45149d2d38f
paper_authors.name: Honggang Chen
paper_authors._id: 68353258d005e45149d2d390
paper_authors.name: Weijia Li
paper_authors._id: 68353258d005e45149d2d391
paper_authors.name: Xuming Hu
paper_authors._id: 68353258d005e45149d2d392
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:22:15.329Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 68353258d005e45149d2d393
paper_authors.name: Linfeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T08:22:07.787Z
paper_authors._id: 642ec9831d1737803dc1c30a
paper_authors.avatarUrl: /avatars/c9ded838bad09004c15a27200e66a108.svg
paper_authors.fullname: linfeng zhang
paper_authors.type: user
paper_authors.user: linfengZ
paper_title: Shifting AI Efficiency From Model-Centric to Data-Centric Compression
paper_summary: The rapid advancement of large language models (LLMs) and multi-modal LLMs
  (MLLMs) has historically relied on model-centric scaling through increasing
  parameter counts from millions to hundreds of billions to drive performance
  gains. However, as we approach hardware limits on model size, the dominant
  computational bottleneck has fundamentally shifted to the quadratic cost of
  self-attention over long token sequences, now driven by ultra-long text
  contexts, high-resolution images, and extended videos. In this position paper,
  we argue that the focus of research for efficient AI is shifting from
  model-centric compression to data-centric compression. We position token
  compression as the new frontier, which improves AI efficiency via reducing the
  number of tokens during model training or inference. Through comprehensive
  analysis, we first examine recent developments in long-context AI across
  various domains and establish a unified mathematical framework for existing
  model efficiency strategies, demonstrating why token compression represents a
  crucial paradigm shift in addressing long-context overhead. Subsequently, we
  systematically review the research landscape of token compression, analyzing
  its fundamental benefits and identifying its compelling advantages across
  diverse scenarios. Furthermore, we provide an in-depth analysis of current
  challenges in token compression research and outline promising future
  directions. Ultimately, our work aims to offer a fresh perspective on AI
  efficiency, synthesize existing research, and catalyze innovative developments
  to address the challenges that increasing context lengths pose to the AI
  community's advancement.
paper_ai_summary: The focus in AI research is shifting from model-centric to data-centric compression, with token compression identified as key to improving efficiency in handling long-context scenarios.
paper_ai_keywords: large language models multi-modal LLMs self-attention token compression long-context AI mathematical framework model efficiency long-context overhead current challenges future directions