title: The Shaped Transformer: Attention Models in the Infinite Depth-and-Width
    Limit
summary: In deep learning theory, the covariance matrix of the representations serves
  as a proxy to examine the network's trainability. Motivated by the success of
  Transformers, we study the covariance matrix of a modified Softmax-based
  attention model with skip connections in the proportional limit of
  infinite-depth-and-width. We show that at initialization the limiting
  distribution can be described by a stochastic differential equation (SDE)
  indexed by the depth-to-width ratio. To achieve a well-defined stochastic
  limit, the Transformer's attention mechanism is modified by centering the
  Softmax output at identity, and scaling the Softmax logits by a width-dependent
  temperature parameter. We examine the stability of the network through the
  corresponding SDE, showing how the scale of both the drift and diffusion can be
  elegantly controlled with the aid of residual connections. The existence of a
  stable SDE implies that the covariance structure is well-behaved, even for very
  large depth and width, thus preventing the notorious issues of rank degeneracy
  in deep attention models. Finally, we show, through simulations, that the SDE
  provides a surprisingly good description of the corresponding finite-size
  model. We coin the name shaped Transformer for these architectural
  modifications.
paper_authors._id: 64a22d6d8f953945fa4b4f69
paper_authors.name: Lorenzo Noci
paper_authors._id: 64a22d6d8f953945fa4b4f6a
paper_authors.name: Chuning Li
paper_authors._id: 64a22d6d8f953945fa4b4f6b
paper_authors.name: Mufan Bill Li
paper_authors._id: 64a22d6d8f953945fa4b4f6c
paper_authors.name: Bobby He
paper_authors._id: 64a22d6d8f953945fa4b4f6d
paper_authors.name: Thomas Hofmann
paper_authors._id: 64a22d6d8f953945fa4b4f6e
paper_authors.name: Chris Maddison
paper_authors._id: 64a22d6d8f953945fa4b4f6f
paper_authors.name: Daniel M. Roy
paper_title: The Shaped Transformer: Attention Models in the Infinite Depth-and-Width
    Limit
paper_summary: In deep learning theory, the covariance matrix of the representations serves
  as a proxy to examine the network's trainability. Motivated by the success of
  Transformers, we study the covariance matrix of a modified Softmax-based
  attention model with skip connections in the proportional limit of
  infinite-depth-and-width. We show that at initialization the limiting
  distribution can be described by a stochastic differential equation (SDE)
  indexed by the depth-to-width ratio. To achieve a well-defined stochastic
  limit, the Transformer's attention mechanism is modified by centering the
  Softmax output at identity, and scaling the Softmax logits by a width-dependent
  temperature parameter. We examine the stability of the network through the
  corresponding SDE, showing how the scale of both the drift and diffusion can be
  elegantly controlled with the aid of residual connections. The existence of a
  stable SDE implies that the covariance structure is well-behaved, even for very
  large depth and width, thus preventing the notorious issues of rank degeneracy
  in deep attention models. Finally, we show, through simulations, that the SDE
  provides a surprisingly good description of the corresponding finite-size
  model. We coin the name shaped Transformer for these architectural
  modifications.
paper_ai_summary: A modified Softmax-based attention model with skip connections exhibits well-behaved covariance structure and stability in the proportional limit of infinite-depth-and-width, preventing rank degeneracy issues.
paper_ai_keywords: covariance matrix representations trainability Transformers Softmax-based attention model skip connections proportional limit infinite-depth-and-width stochastic differential equation SDE depth-to-width ratio centering width-dependent temperature parameter residual connections stability rank degeneracy finite-size model shaped Transformer