title: Token Reduction Should Go Beyond Efficiency in Generative Models -- From
    Vision, Language to Multimodality
summary: In Transformer architectures, tokens\textemdash discrete units derived from
  raw data\textemdash are formed by segmenting inputs into fixed-length chunks.
  Each token is then mapped to an embedding, enabling parallel attention
  computations while preserving the input's essential information. Due to the
  quadratic computational complexity of transformer self-attention mechanisms,
  token reduction has primarily been used as an efficiency strategy. This is
  especially true in single vision and language domains, where it helps balance
  computational costs, memory usage, and inference latency. Despite these
  advances, this paper argues that token reduction should transcend its
  traditional efficiency-oriented role in the era of large generative models.
  Instead, we position it as a fundamental principle in generative modeling,
  critically influencing both model architecture and broader applications.
  Specifically, we contend that across vision, language, and multimodal systems,
  token reduction can: (i) facilitate deeper multimodal integration and
  alignment, (ii) mitigate "overthinking" and hallucinations, (iii) maintain
  coherence over long inputs, and (iv) enhance training stability, etc. We
  reframe token reduction as more than an efficiency measure. By doing so, we
  outline promising future directions, including algorithm design, reinforcement
  learning-guided token reduction, token optimization for in-context learning,
  and broader ML and scientific domains. We highlight its potential to drive new
  model architectures and learning strategies that improve robustness, increase
  interpretability, and better align with the objectives of generative modeling.
paper_authors._id: 683866e4ae92c7d496fdd866
paper_authors.name: Zhenglun Kong
paper_authors._id: 683866e4ae92c7d496fdd867
paper_authors.name: Yize Li
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-29T15:39:45.735Z
paper_authors._id: 6552d3f69fa2f7205a1f94a2
paper_authors.avatarUrl: /avatars/8b857fe89586b25b54864da91ce1b35a.svg
paper_authors.fullname: Yize Li
paper_authors.type: user
paper_authors.user: yeezlee
paper_authors._id: 683866e4ae92c7d496fdd868
paper_authors.name: Fanhu Zeng
paper_authors._id: 683866e4ae92c7d496fdd869
paper_authors.name: Lei Xin
paper_authors._id: 683866e4ae92c7d496fdd86a
paper_authors.name: Shvat Messica
paper_authors._id: 683866e4ae92c7d496fdd86b
paper_authors.name: Xue Lin
paper_authors._id: 683866e4ae92c7d496fdd86c
paper_authors.name: Pu Zhao
paper_authors._id: 683866e4ae92c7d496fdd86d
paper_authors.name: Manolis Kellis
paper_authors._id: 683866e4ae92c7d496fdd86e
paper_authors.name: Hao Tang
paper_authors._id: 683866e4ae92c7d496fdd86f
paper_authors.name: Marinka Zitnik
paper_title: Token Reduction Should Go Beyond Efficiency in Generative Models -- From
    Vision, Language to Multimodality
paper_summary: In Transformer architectures, tokens\textemdash discrete units derived from
  raw data\textemdash are formed by segmenting inputs into fixed-length chunks.
  Each token is then mapped to an embedding, enabling parallel attention
  computations while preserving the input's essential information. Due to the
  quadratic computational complexity of transformer self-attention mechanisms,
  token reduction has primarily been used as an efficiency strategy. This is
  especially true in single vision and language domains, where it helps balance
  computational costs, memory usage, and inference latency. Despite these
  advances, this paper argues that token reduction should transcend its
  traditional efficiency-oriented role in the era of large generative models.
  Instead, we position it as a fundamental principle in generative modeling,
  critically influencing both model architecture and broader applications.
  Specifically, we contend that across vision, language, and multimodal systems,
  token reduction can: (i) facilitate deeper multimodal integration and
  alignment, (ii) mitigate "overthinking" and hallucinations, (iii) maintain
  coherence over long inputs, and (iv) enhance training stability, etc. We
  reframe token reduction as more than an efficiency measure. By doing so, we
  outline promising future directions, including algorithm design, reinforcement
  learning-guided token reduction, token optimization for in-context learning,
  and broader ML and scientific domains. We highlight its potential to drive new
  model architectures and learning strategies that improve robustness, increase
  interpretability, and better align with the objectives of generative modeling.
paper_ai_summary: Token reduction in Transformer models, beyond efficiency, enhances multimodal integration, reduces hallucinations, and improves training stability in generative modeling.
paper_ai_keywords: Transformer architectures token reduction self-attention mechanisms multimodal integration hallucinations training stability generative modeling