Answers.Body: <p>Weights are not normally shared across Transformer layers in vanilla Transformers. However, there has been research done in testing out sharing weights, and sometimes they improve the scores. Here are some examples:</p>
  <p><a href="https://arxiv.org/pdf/1909.11942.pdf" rel="nofollow noreferrer">ALBERT</a> is an improvement on BERT (so only uses the encoding side, no decoder), and shows that sharing the attention weights only <span class="math-container">$\left\{ W_i^Q, W_i^K, W_i^V \right\}$</span> across all Transformer layers for large networks either result in the same accuracy or slightly improved accuracy, while significantly reducing model size. Sharing the position-wise FFN layer though hindered performance.</p>
  <p><a href="https://arxiv.org/pdf/1910.10683.pdf" rel="nofollow noreferrer">Text-to-Text Transfer Transformer</a> shows that sharing the weights between encoder and decoder layer of the transformer (so e.g. layer 1 encoding weights = layer 1 decoding weights) barely affected accuracy (it dropped by 0.5%), but the model size is halved.</p>
  <p>I'm sure there are more papers I have forgotten. Sharing weights is still an active area of research, but for vanilla transformers it is assumed they do not share weights.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-17T01:45:18.143Z
Answers.LastActivityDate: 2020-12-17T01:45:18.143Z
Answers.UserDisplayName: user3667125
Body: <p>In articles that describe neural architectures with multiple attention layers of the same form, are the weight matrices usually the same across the layers?  Consider as an example, &quot;Attention is all you need&quot;. The authors stack several layers of multi-head self-attention in which each layer has the same number of heads.  Each head <span class="math-container">$i$</span> involves a trainable weight matrix <span class="math-container">$W_{i}^{Q}$</span>.  There is no subscript, superscript, or any other indication that this matrix is different for each layer.  My questions is this: are there separate <span class="math-container">$W_{i}^{Q}$</span> for layers <span class="math-container">$1,2,3,...$</span> or is this a single matrix shared throughout layers?</p>
  <p>My intuition is that the authors of the paper wanted to cut down on notation, and that the matrices are different in different layers.  But I want to be sure I understand this, since I see the same kind of thing in many other papers as well.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-16T23:06:47.517Z
Comments.Text: The official implementation of the initial transformer should have been done in `tensor2tensor`, as stated in the paper (if I remember correctly: please, check the footnotes of the paper, I think they say it there somewhere). Maybe it's this one: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer.py. You can check what they did there. You can then later provide an answer to your own question below ;)
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-16T23:25:53.580Z
Comments.Text: It's a good idea, but one I already tried and didn't succeed at.  I tried to read the code for both TensorFlow and PyTorch, and couldn't figure it out.
Comments.UserDisplayName: BioBroo
Tags: <transformer><attention><weights>
Title: In attention models with multiple layers, are weight matrices shared across layers?
UserDisplayName: BioBroo
ThreadText: # In attention models with multiple layers, are weight matrices shared across layers?
  
  (Asked by: username_0 on 2020-12-16)
  
  In articles that describe neural architectures with multiple attention layers of the same form, are the weight matrices usually the same across the layers? Consider as an example, "Attention is all you need". The authors stack several layers of multi-head self-attention in which each layer has the same number of heads. Each head $i$ involves a trainable weight matrix $W\_{i}^{Q}$. There is no subscript, superscript, or any other indication that this matrix is different for each layer. My questions is this: are there separate $W\_{i}^{Q}$ for layers $1,2,3,...$ or is this a single matrix shared throughout layers?
  
  My intuition is that the authors of the paper wanted to cut down on notation, and that the matrices are different in different layers. But I want to be sure I understand this, since I see the same kind of thing in many other papers as well.
  
  **Tags:** transformer, attention, weights
  > **Comments:**
  > **username_1**: The official implementation of the initial transformer should have been done in `tensor2tensor`, as stated in the paper (if I remember correctly: please, check the footnotes of the paper, I think they say it there somewhere). Maybe it's this one: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer.py. You can check what they did there. You can then later provide an answer to your own question below ;)  
  > **username_0**: It's a good idea, but one I already tried and didn't succeed at.  I tried to read the code for both TensorFlow and PyTorch, and couldn't figure it out.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-12-17. Score: 1 | Accepted answer
  
  Weights are not normally shared across Transformer layers in vanilla Transformers. However, there has been research done in testing out sharing weights, and sometimes they improve the scores. Here are some examples:
  
  [ALBERT](https://arxiv.org/pdf/1909.11942.pdf) is an improvement on BERT (so only uses the encoding side, no decoder), and shows that sharing the attention weights only $\left\{ W\_i^Q, W\_i^K, W\_i^V \right\}$ across all Transformer layers for large networks either result in the same accuracy or slightly improved accuracy, while significantly reducing model size. Sharing the position-wise FFN layer though hindered performance.
  
  [Text-to-Text Transfer Transformer](https://arxiv.org/pdf/1910.10683.pdf) shows that sharing the weights between encoder and decoder layer of the transformer (so e.g. layer 1 encoding weights = layer 1 decoding weights) barely affected accuracy (it dropped by 0.5%), but the model size is halved.
  
  I'm sure there are more papers I have forgotten. Sharing weights is still an active area of research, but for vanilla transformers it is assumed they do not share weights.
  
  
  ---
  
  