Body: <p>Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at <a href="https://arxiv.org/pdf/1506.07503.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1506.07503.pdf</a>. (it seems this attention is used in various forms of GNMT and text-to-speech sythesizers like tacotron-2 <a href="https://github.com/Rayhane-mamah/Tacotron-2" rel="nofollow noreferrer">https://github.com/Rayhane-mamah/Tacotron-2</a>).</p>
  
  <p>Even after repeated readings of this paper and other articles about Attention-mechanism, I'm confused about the dimensions of the matrices used, as the paper doesn't seem to describe it. My understanding is:</p>
  
  <ul>
  <li><p>If I have decoder hidden dim 1024, that means <span class="math-container">${s_{i-1}}$</span> vector is 1024 length.</p></li>
  <li><p>If I have encoder output dim 512, that means <span class="math-container">$h_{j}$</span> vector is 512 length.</p></li>
  <li><p>If total inputs to encoder is 256, then number of <span class="math-container">$j$</span> can be from 1 to 256.</p></li>
  <li><p>Since <span class="math-container">$W x S_{i-1}$</span> is a matrix multiply, it seems <span class="math-container">$cols(W)$</span> should match <span class="math-container">$rows(S_{i-1})$</span>, but <span class="math-container">$rows(W)$</span> still remain undefined. Same seems true for matrices <span class="math-container">$V, U, w, b$</span>.</p></li>
  </ul>
  
  <p>This is page-3/4 from the paper above that describes Attention-layer:</p>
  
  <p><a href="https://i.sstatic.net/xKD2U.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xKD2U.png" alt="enter image description here"></a></p>
  
  <p>I'm unsure how to make sense of this. Am I missing something, or can someone explain this?</p>
  
  <p>What I don't understand is:</p>
  
  <ul>
  <li><p>What is the dimension of previous alignment (denoted by <span class="math-container">$alpha_{i-1})$</span>? Shouldn't it be total values of <span class="math-container">$j$</span> in <span class="math-container">$h_{j}$</span> (which is 256 and means total different encoder output states)?</p></li>
  <li><p>What is the dimension of <span class="math-container">$f_{i,j}$</span> and convolution filter <span class="math-container">$F$</span>? (the paper says <span class="math-container">$F$</span> belongs to <span class="math-container">$kxr$</span> shape but doesn't define <span class="math-container">$'r'$</span> anywhere). What is <span class="math-container">$'r'$</span> and what does <span class="math-container">$'k x r'$</span> mean here?</p></li>
  <li><p>How are these unknown dimensions for matrices <span class="math-container">$'V, U, w, b'$</span> described above determined in this model?</p></li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-04T14:38:50.740Z
Comments.Text: I suggest you ask only one question per post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-04T15:29:19.767Z
Comments.Text: @nbro these are very related questions, and it seems answers to anyone is sort of dependent on others too.
Comments.UserDisplayName: Joe Black
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-04T15:29:25.980Z
Comments.Text: i didn't receive or see any answer to these questions, so figured it'd be better if they are all together so if anyone who knows the answers to these can answer them in one go.
Comments.UserDisplayName: Joe Black
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-04T15:47:16.653Z
Comments.Text: Only someone that has read the related papers and is an expert on the topic could answer such questions. But there aren't many experts on specific topics. That's why I am suggesting you split this post into simpler ones.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-29T16:06:00.233Z
Comments.Text: Unfortunately none of the many tutorials explain the concept. Since even the original paper does not explain it, the best approach may just be to ask the authors.
Comments.UserDisplayName: Nav
Tags: <deep-learning><recurrent-neural-networks><long-short-term-memory><transformer><attention>
Title: How to understand the matrices used in the Attention layer?
UserDisplayName: Joe Black
ThreadText: # How to understand the matrices used in the Attention layer?
  
  (Asked by: username_0 on 2020-06-02)
  
  Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at <https://arxiv.org/pdf/1506.07503.pdf>. (it seems this attention is used in various forms of GNMT and text-to-speech sythesizers like tacotron-2 <https://github.com/Rayhane-mamah/Tacotron-2>).
  
  Even after repeated readings of this paper and other articles about Attention-mechanism, I'm confused about the dimensions of the matrices used, as the paper doesn't seem to describe it. My understanding is:
  
  * If I have decoder hidden dim 1024, that means ${s\_{i-1}}$ vector is 1024 length.
  * If I have encoder output dim 512, that means $h\_{j}$ vector is 512 length.
  * If total inputs to encoder is 256, then number of $j$ can be from 1 to 256.
  * Since $W x S\_{i-1}$ is a matrix multiply, it seems $cols(W)$ should match $rows(S\_{i-1})$, but $rows(W)$ still remain undefined. Same seems true for matrices $V, U, w, b$.
  
  This is page-3/4 from the paper above that describes Attention-layer:
  
  [![enter image description here](https://i.sstatic.net/xKD2U.png)](https://i.sstatic.net/xKD2U.png)
  
  I'm unsure how to make sense of this. Am I missing something, or can someone explain this?
  
  What I don't understand is:
  
  * What is the dimension of previous alignment (denoted by $alpha\_{i-1})$? Shouldn't it be total values of $j$ in $h\_{j}$ (which is 256 and means total different encoder output states)?
  * What is the dimension of $f\_{i,j}$ and convolution filter $F$? (the paper says $F$ belongs to $kxr$ shape but doesn't define $'r'$ anywhere). What is $'r'$ and what does $'k x r'$ mean here?
  * How are these unknown dimensions for matrices $'V, U, w, b'$ described above determined in this model?
  
  **Tags:** deep-learning, recurrent-neural-networks, long-short-term-memory, transformer, attention
  
  > **Comments:**
  > **username_1**: I suggest you ask only one question per post.  
  > **username_0**: @username_1 these are very related questions, and it seems answers to anyone is sort of dependent on others too.  
  > **username_0**: i didn't receive or see any answer to these questions, so figured it'd be better if they are all together so if anyone who knows the answers to these can answer them in one go.  
  > **username_1**: Only someone that has read the related papers and is an expert on the topic could answer such questions. But there aren't many experts on specific topics. That's why I am suggesting you split this post into simpler ones.  
  > **username_2**: Unfortunately none of the many tutorials explain the concept. Since even the original paper does not explain it, the best approach may just be to ask the authors.  
  
  ---
  
  ## 0 answers
  
  