Token Reduction Should Go Beyond Efficiency in Generative Models -- From
  Vision, Language to Multimodality
In Transformer architectures, tokens\textemdash discrete units derived from
raw data\textemdash are formed by segmenting inputs into fixed-length chunks.
Each token is then mapped to an embedding, enabling parallel attention
computations while preserving the input's essential information. Due to the
quadratic computational complexity of transformer self-attention mechanisms,
token reduction has primarily been used as an efficiency strategy. This is
especially true in single vision and language domains, where it helps balance
computational costs, memory usage, and inference latency. Despite these
advances, this paper argues that token reduction should transcend its
traditional efficiency-oriented role in the era of large generative models.
Instead, we position it as a fundamental principle in generative modeling,
critically influencing both model architecture and broader applications.
Specifically, we contend that across vision, language, and multimodal systems,
token reduction can: (i) facilitate deeper multimodal integration and
alignment, (ii) mitigate "overthinking" and hallucinations, (iii) maintain
coherence over long inputs, and (iv) enhance training stability, etc. We
reframe token reduction as more than an efficiency measure. By doing so, we
outline promising future directions, including algorithm design, reinforcement
learning-guided token reduction, token optimization for in-context learning,
and broader ML and scientific domains. We highlight its potential to drive new
model architectures and learning strategies that improve robustness, increase
interpretability, and better align with the objectives of generative modeling.
683866e4ae92c7d496fdd866
Zhenglun Kong
683866e4ae92c7d496fdd867
Yize Li
extracted_confirmed
2025-05-29T15:39:45.735Z
6552d3f69fa2f7205a1f94a2
/avatars/8b857fe89586b25b54864da91ce1b35a.svg
Yize Li
user
yeezlee
683866e4ae92c7d496fdd868
Fanhu Zeng
683866e4ae92c7d496fdd869
Lei Xin
683866e4ae92c7d496fdd86a
Shvat Messica
683866e4ae92c7d496fdd86b
Xue Lin
683866e4ae92c7d496fdd86c
Pu Zhao
683866e4ae92c7d496fdd86d
Manolis Kellis
683866e4ae92c7d496fdd86e
Hao Tang
683866e4ae92c7d496fdd86f
Marinka Zitnik
Token Reduction Should Go Beyond Efficiency in Generative Models -- From
  Vision, Language to Multimodality
In Transformer architectures, tokens\textemdash discrete units derived from
raw data\textemdash are formed by segmenting inputs into fixed-length chunks.
Each token is then mapped to an embedding, enabling parallel attention
computations while preserving the input's essential information. Due to the
quadratic computational complexity of transformer self-attention mechanisms,
token reduction has primarily been used as an efficiency strategy. This is
especially true in single vision and language domains, where it helps balance
computational costs, memory usage, and inference latency. Despite these
advances, this paper argues that token reduction should transcend its
traditional efficiency-oriented role in the era of large generative models.
Instead, we position it as a fundamental principle in generative modeling,
critically influencing both model architecture and broader applications.
Specifically, we contend that across vision, language, and multimodal systems,
token reduction can: (i) facilitate deeper multimodal integration and
alignment, (ii) mitigate "overthinking" and hallucinations, (iii) maintain
coherence over long inputs, and (iv) enhance training stability, etc. We
reframe token reduction as more than an efficiency measure. By doing so, we
outline promising future directions, including algorithm design, reinforcement
learning-guided token reduction, token optimization for in-context learning,
and broader ML and scientific domains. We highlight its potential to drive new
model architectures and learning strategies that improve robustness, increase
interpretability, and better align with the objectives of generative modeling.
Token reduction in Transformer models, beyond efficiency, enhances multimodal integration, reduces hallucinations, and improves training stability in generative modeling.
Transformer architectures token reduction self-attention mechanisms multimodal integration hallucinations training stability generative modeling