title: Scalable-Softmax Is Superior for Attention
summary: The maximum element of the vector output by the Softmax function approaches
  zero as the input vector size increases. Transformer-based language models rely
  on Softmax to compute attention scores, causing the attention distribution to
  flatten as the context size grows. This reduces the model's ability to
  prioritize key information effectively and potentially limits its length
  generalization. To address this problem, we propose Scalable-Softmax (SSMax),
  which replaces Softmax in scenarios where the input vector size varies. SSMax
  can be seamlessly integrated into existing Transformer-based architectures.
  Experimental results in language modeling show that models using SSMax not only
  achieve faster loss reduction during pretraining but also significantly improve
  performance in long contexts and key information retrieval. Furthermore, an
  analysis of attention scores reveals that SSMax enables the model to focus
  attention on key information even in long contexts. Additionally, although
  models that use SSMax from the beginning of pretraining achieve better length
  generalization, those that have already started pretraining can still gain some
  of this ability by replacing Softmax in the attention layers with SSMax, either
  during or after pretraining.
paper_authors._id: 67a0e9707ddf31accd7b2510
paper_authors.name: Ken M. Nakanishi
paper_title: Scalable-Softmax Is Superior for Attention
paper_summary: The maximum element of the vector output by the Softmax function approaches
  zero as the input vector size increases. Transformer-based language models rely
  on Softmax to compute attention scores, causing the attention distribution to
  flatten as the context size grows. This reduces the model's ability to
  prioritize key information effectively and potentially limits its length
  generalization. To address this problem, we propose Scalable-Softmax (SSMax),
  which replaces Softmax in scenarios where the input vector size varies. SSMax
  can be seamlessly integrated into existing Transformer-based architectures.
  Experimental results in language modeling show that models using SSMax not only
  achieve faster loss reduction during pretraining but also significantly improve
  performance in long contexts and key information retrieval. Furthermore, an
  analysis of attention scores reveals that SSMax enables the model to focus
  attention on key information even in long contexts. Additionally, although
  models that use SSMax from the beginning of pretraining achieve better length
  generalization, those that have already started pretraining can still gain some
  of this ability by replacing Softmax in the attention layers with SSMax, either
  during or after pretraining.
paper_ai_summary: Scalable-Softmax (SSMax) enhances Transformer-based models by improving attention distribution and long-context performance, resolving issues posed by the Softmax function.
paper_ai_keywords: Softmax attention scores context size Scalable-Softmax (SSMax) attention distribution long contexts key information retrieval pretraining length generalization