title: TokenFormer: Rethinking Transformer Scaling with Tokenized Model
    Parameters
summary: Transformers have become the predominant architecture in foundation models
  due to their excellent performance across various domains. However, the
  substantial cost of scaling these models remains a significant concern. This
  problem arises primarily from their dependence on a fixed number of parameters
  within linear projections. When architectural modifications (e.g., channel
  dimensions) are introduced, the entire model typically requires retraining from
  scratch. As model sizes continue growing, this strategy results in increasingly
  high computational costs and becomes unsustainable. To overcome this problem,
  we introduce TokenFormer, a natively scalable architecture that leverages the
  attention mechanism not only for computations among input tokens but also for
  interactions between tokens and model parameters, thereby enhancing
  architectural flexibility. By treating model parameters as tokens, we replace
  all the linear projections in Transformers with our token-parameter attention
  layer, where input tokens act as queries and model parameters as keys and
  values. This reformulation allows for progressive and efficient scaling without
  necessitating retraining from scratch. Our model scales from 124M to 1.4B
  parameters by incrementally adding new key-value parameter pairs, achieving
  performance comparable to Transformers trained from scratch while greatly
  reducing training costs. Code and models are available at
  https://github.com/Haiyang-W/TokenFormer.
paper_authors._id: 6722db7db23a243a937ee098
paper_authors.name: Haiyang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-31T14:06:06.262Z
paper_authors._id: 65f43c3cc9940817caaf4434
paper_authors.avatarUrl: /avatars/ecec2856ba7a7d3421a2071a0a88800b.svg
paper_authors.fullname: Haiyang Wang
paper_authors.type: user
paper_authors.user: Haiyang-W
paper_authors._id: 6722db7db23a243a937ee099
paper_authors.name: Yue Fan
paper_authors._id: 6722db7db23a243a937ee09a
paper_authors.name: Muhammad Ferjad Naeem
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T15:37:48.498Z
paper_authors._id: 67bf33a512368ec2fad4fe29
paper_authors.avatarUrl: /avatars/ea5c03744ec1c2bcc0e6c13efc8f7ddc.svg
paper_authors.fullname: Muhammad Ferjad Naeem
paper_authors.type: user
paper_authors.user: ferjad
paper_authors._id: 6722db7db23a243a937ee09b
paper_authors.name: Yongqin Xian
paper_authors._id: 6722db7db23a243a937ee09c
paper_authors.name: Jan Eric Lenssen
paper_authors._id: 6722db7db23a243a937ee09d
paper_authors.name: Liwei Wang
paper_authors._id: 6722db7db23a243a937ee09e
paper_authors.name: Federico Tombari
paper_authors._id: 6722db7db23a243a937ee09f
paper_authors.name: Bernt Schiele
paper_title: TokenFormer: Rethinking Transformer Scaling with Tokenized Model
    Parameters
paper_summary: Transformers have become the predominant architecture in foundation models
  due to their excellent performance across various domains. However, the
  substantial cost of scaling these models remains a significant concern. This
  problem arises primarily from their dependence on a fixed number of parameters
  within linear projections. When architectural modifications (e.g., channel
  dimensions) are introduced, the entire model typically requires retraining from
  scratch. As model sizes continue growing, this strategy results in increasingly
  high computational costs and becomes unsustainable. To overcome this problem,
  we introduce TokenFormer, a natively scalable architecture that leverages the
  attention mechanism not only for computations among input tokens but also for
  interactions between tokens and model parameters, thereby enhancing
  architectural flexibility. By treating model parameters as tokens, we replace
  all the linear projections in Transformers with our token-parameter attention
  layer, where input tokens act as queries and model parameters as keys and
  values. This reformulation allows for progressive and efficient scaling without
  necessitating retraining from scratch. Our model scales from 124M to 1.4B
  parameters by incrementally adding new key-value parameter pairs, achieving
  performance comparable to Transformers trained from scratch while greatly
  reducing training costs. Code and models are available at
  https://github.com/Haiyang-W/TokenFormer.
paper_ai_keywords: Transformers foundation models scaling parameters linear projections architectural modifications retraining from scratch natively scalable architecture attention uncertainty TokenFormer token-parameter attention queries keys values progressive and efficient scaling training costs performance retraining from scratch F input tokens key-value parameter pairs incremental adding v parameter pairs v metric vB parameters vM parameters