Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing
  Important Tokens
Transformer models are foundational to natural language processing (NLP) and
computer vision. Despite various recent works devoted to reducing the quadratic
cost of such models (as a function of the sequence length n), dealing with
ultra long sequences efficiently (e.g., with more than 16K tokens) remains
challenging. Applications such as answering questions based on an entire book
or summarizing a scientific article are inefficient or infeasible. In this
paper, we propose to significantly reduce the dependency of a Transformer
model's complexity on n, by compressing the input into a representation whose
size r is independent of n at each layer. Specifically, by exploiting the
fact that in many tasks, only a small subset of special tokens (we call
VIP-tokens) are most relevant to the final prediction, we propose a VIP-token
centric compression (Vcc) scheme which selectively compresses the input
sequence based on their impact on approximating the representation of these
VIP-tokens. Compared with competitive baselines, the proposed algorithm not
only is efficient (achieving more than 3times efficiency improvement
compared to baselines on 4K and 16K lengths), but also achieves competitive or
better performance on a large number of tasks. Further, we show that our
algorithm can be scaled to 128K tokens (or more) while consistently offering
accuracy improvement.
6459adf135cfaca4645c8005
Zhanpeng Zeng
extracted_confirmed
2023-05-24T15:34:09.568Z
646e2ddf4ad7f90727a0658b
/avatars/f62a9f835279b49ca5899c3aa6228dca.svg
Zhanpeng Zeng
user
mlpen
6459adf135cfaca4645c8006
Cole Hawkins
extracted_pending
2024-07-02T15:45:26.677Z
66841ee5c1be1cd1690b11f9
/avatars/9964b92318bf4dd4ec0142a03a321a8e.svg
Cole Hawkins
user
colehawk
6459adf135cfaca4645c8007
Mingyi Hong
extracted_confirmed
2025-05-31T18:34:23.621Z
679a5fa807076ced8ae5f41b
/avatars/63cac0a05c9c7500c9ab9c75667a38d7.svg
Mingyi Hong
user
poborskypaul
6459adf135cfaca4645c8008
Aston Zhang
6459adf135cfaca4645c8009
Nikolaos Pappas
extracted_pending
2024-08-09T00:55:43.928Z
66b568fc8de19c0bb34a67a3
/avatars/5d5930e11f5b1cd7e66eb3bfa43e9a22.svg
Nikos Pappas
user
nppappa
6459adf135cfaca4645c800a
Vikas Singh
6459adf135cfaca4645c800b
Shuai Zheng
Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing
  Important Tokens
Transformer models are foundational to natural language processing (NLP) and
computer vision. Despite various recent works devoted to reducing the quadratic
cost of such models (as a function of the sequence length n), dealing with
ultra long sequences efficiently (e.g., with more than 16K tokens) remains
challenging. Applications such as answering questions based on an entire book
or summarizing a scientific article are inefficient or infeasible. In this
paper, we propose to significantly reduce the dependency of a Transformer
model's complexity on n, by compressing the input into a representation whose
size r is independent of n at each layer. Specifically, by exploiting the
fact that in many tasks, only a small subset of special tokens (we call
VIP-tokens) are most relevant to the final prediction, we propose a VIP-token
centric compression (Vcc) scheme which selectively compresses the input
sequence based on their impact on approximating the representation of these
VIP-tokens. Compared with competitive baselines, the proposed algorithm not
only is efficient (achieving more than 3times efficiency improvement
compared to baselines on 4K and 16K lengths), but also achieves competitive or
better performance on a large number of tasks. Further, we show that our
algorithm can be scaled to 128K tokens (or more) while consistently offering
accuracy improvement.
A VIP-token centric compression scheme for Transformer models significantly enhances efficiency and scalability in handling ultra long sequences.
Transformer models VIP-tokens VIP-token centric compression Vcc scheme sequence length efficient scalable ultra long sequences