title: Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing
    Important Tokens
summary: Transformer models are foundational to natural language processing (NLP) and
  computer vision. Despite various recent works devoted to reducing the quadratic
  cost of such models (as a function of the sequence length n), dealing with
  ultra long sequences efficiently (e.g., with more than 16K tokens) remains
  challenging. Applications such as answering questions based on an entire book
  or summarizing a scientific article are inefficient or infeasible. In this
  paper, we propose to significantly reduce the dependency of a Transformer
  model's complexity on n, by compressing the input into a representation whose
  size r is independent of n at each layer. Specifically, by exploiting the
  fact that in many tasks, only a small subset of special tokens (we call
  VIP-tokens) are most relevant to the final prediction, we propose a VIP-token
  centric compression (Vcc) scheme which selectively compresses the input
  sequence based on their impact on approximating the representation of these
  VIP-tokens. Compared with competitive baselines, the proposed algorithm not
  only is efficient (achieving more than 3times efficiency improvement
  compared to baselines on 4K and 16K lengths), but also achieves competitive or
  better performance on a large number of tasks. Further, we show that our
  algorithm can be scaled to 128K tokens (or more) while consistently offering
  accuracy improvement.
paper_authors._id: 6459adf135cfaca4645c8005
paper_authors.name: Zhanpeng Zeng
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-24T15:34:09.568Z
paper_authors._id: 646e2ddf4ad7f90727a0658b
paper_authors.avatarUrl: /avatars/f62a9f835279b49ca5899c3aa6228dca.svg
paper_authors.fullname: Zhanpeng Zeng
paper_authors.type: user
paper_authors.user: mlpen
paper_authors._id: 6459adf135cfaca4645c8006
paper_authors.name: Cole Hawkins
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-07-02T15:45:26.677Z
paper_authors._id: 66841ee5c1be1cd1690b11f9
paper_authors.avatarUrl: /avatars/9964b92318bf4dd4ec0142a03a321a8e.svg
paper_authors.fullname: Cole Hawkins
paper_authors.type: user
paper_authors.user: colehawk
paper_authors._id: 6459adf135cfaca4645c8007
paper_authors.name: Mingyi Hong
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-31T18:34:23.621Z
paper_authors._id: 679a5fa807076ced8ae5f41b
paper_authors.avatarUrl: /avatars/63cac0a05c9c7500c9ab9c75667a38d7.svg
paper_authors.fullname: Mingyi Hong
paper_authors.type: user
paper_authors.user: poborskypaul
paper_authors._id: 6459adf135cfaca4645c8008
paper_authors.name: Aston Zhang
paper_authors._id: 6459adf135cfaca4645c8009
paper_authors.name: Nikolaos Pappas
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-09T00:55:43.928Z
paper_authors._id: 66b568fc8de19c0bb34a67a3
paper_authors.avatarUrl: /avatars/5d5930e11f5b1cd7e66eb3bfa43e9a22.svg
paper_authors.fullname: Nikos Pappas
paper_authors.type: user
paper_authors.user: nppappa
paper_authors._id: 6459adf135cfaca4645c800a
paper_authors.name: Vikas Singh
paper_authors._id: 6459adf135cfaca4645c800b
paper_authors.name: Shuai Zheng
paper_title: Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing
    Important Tokens
paper_summary: Transformer models are foundational to natural language processing (NLP) and
  computer vision. Despite various recent works devoted to reducing the quadratic
  cost of such models (as a function of the sequence length n), dealing with
  ultra long sequences efficiently (e.g., with more than 16K tokens) remains
  challenging. Applications such as answering questions based on an entire book
  or summarizing a scientific article are inefficient or infeasible. In this
  paper, we propose to significantly reduce the dependency of a Transformer
  model's complexity on n, by compressing the input into a representation whose
  size r is independent of n at each layer. Specifically, by exploiting the
  fact that in many tasks, only a small subset of special tokens (we call
  VIP-tokens) are most relevant to the final prediction, we propose a VIP-token
  centric compression (Vcc) scheme which selectively compresses the input
  sequence based on their impact on approximating the representation of these
  VIP-tokens. Compared with competitive baselines, the proposed algorithm not
  only is efficient (achieving more than 3times efficiency improvement
  compared to baselines on 4K and 16K lengths), but also achieves competitive or
  better performance on a large number of tasks. Further, we show that our
  algorithm can be scaled to 128K tokens (or more) while consistently offering
  accuracy improvement.
paper_ai_summary: A VIP-token centric compression scheme for Transformer models significantly enhances efficiency and scalability in handling ultra long sequences.
paper_ai_keywords: Transformer models VIP-tokens VIP-token centric compression Vcc scheme sequence length efficient scalable ultra long sequences