Answers.Body: <h3>Alignment:</h3>
  <p>We all know that a good translation cannot be done just by splitting words, converting them, and concatenating them back. Otherwise, a dictionary would be just enough. One translation problem is on the <strong>alignment</strong> of the words.
  For example:</p>
  <pre><code>Uma maçã grande e vermelha
  (1)  (2)  (3)  (4)  (5)
   |      \ /   ______/
   |       X   /
   |      / \ /
   |     /   X
   |    /   / \
  (1) (3) (5)  (2)
   A  big red apple
  </code></pre>
  <h3>RNN</h3>
  <p>This article starts by showing how a RNN translator works and their underlying difficulties. And alignment is a huge pain for RNN because either you'd need another method* to solve it, so RNN could focus on smaller tasks each time.
  *And this method usually requires a labeled dataset (like the example above), which is quite tedious to create.</p>
  <p>What if, instead of hacking an external element to guess the alignment, we could just send the whole text and train a single neural network to both:</p>
  <ul>
  <li>Somehow solve the alignment problem.</li>
  <li>Use that to predict the next word.</li>
  </ul>
  <p>Wouldn't that be awesome? Introducing:</p>
  <h3>Transformers</h3>
  <p>They have a built-in self-attention component that scores all previous words according to their relevance for the next translated word!</p>
  <h4>TL;DR:</h4>
  <p>Transformers will automatically solve alignment while translating.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-26T22:05:59.830Z
Answers.LastActivityDate: 2024-06-25T06:13:53.530Z
Answers.LastEditDate: 2024-06-25T06:13:53.530Z
Answers.UserDisplayName: Andre Goulart
Body: <p>I've been reading about transformers &amp; have been having some difficulty understanding the concept of <em>alignment</em>.</p>
  <p>Based on this <a href="https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3#16cb" rel="nofollow noreferrer">article</a></p>
  <blockquote>
  <p>Alignment means matching segments of original text with their corresponding segments of the translation.</p>
  </blockquote>
  <p>Does this mean that, with transformers, we're adding the fully translated sentences as inputs too? What's the purpose of alignment? How exactly do these models figure out how to match the different segments together? I'm pretty sure there's some underlying assumption/knowledge that I'm not fully getting -- but I'm not entirely sure what.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-10T16:37:28.760Z
Comments.Text: It's basically: how do the output neurons that are supposed to output "économique" know to connect to the input neurons that see the word "economic"? We could just connect every output position to every input position but then it would have a small maximum size limit and wouldn't be translationally invariant because the weights would record the exact positions (output word 5 -> input word 2).
Comments.UserDisplayName: Criticize SE actions means ban
Tags: <neural-networks><transformer><attention><machine-translation>
Title: What is the purpose of "alignment" in the self-attention mechanism of transformers?
UserDisplayName: codinggirl123
ThreadText: # What is the purpose of "alignment" in the self-attention mechanism of transformers?
  
  (Asked by: username_0 on 2021-02-04)
  
  I've been reading about transformers & have been having some difficulty understanding the concept of *alignment*.
  
  Based on this [article](https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3#16cb)
  
  > Alignment means matching segments of original text with their corresponding segments of the translation.
  
  Does this mean that, with transformers, we're adding the fully translated sentences as inputs too? What's the purpose of alignment? How exactly do these models figure out how to match the different segments together? I'm pretty sure there's some underlying assumption/knowledge that I'm not fully getting -- but I'm not entirely sure what.
  
  **Tags:** neural-networks, transformer, attention, machine-translation
  
  > **Comments:**
  > **username_1**: It's basically: how do the output neurons that are supposed to output "économique" know to connect to the input neurons that see the word "economic"? We could just connect every output position to every input position but then it would have a small maximum size limit and wouldn't be translationally invariant because the weights would record the exact positions (output word 5 -> input word 2).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-08-26. Score: 1
  
  ### Alignment:
  
  We all know that a good translation cannot be done just by splitting words, converting them, and concatenating them back. Otherwise, a dictionary would be just enough. One translation problem is on the **alignment** of the words.
  For example:
  
  ```
  Uma maçã grande e vermelha
  (1)  (2)  (3)  (4)  (5)
   |      \ /   ______/
   |       X   /
   |      / \ /
   |     /   X
   |    /   / \
  (1) (3) (5)  (2)
   A  big red apple
  
  ```
  
  ### RNN
  
  This article starts by showing how a RNN translator works and their underlying difficulties. And alignment is a huge pain for RNN because either you'd need another method\* to solve it, so RNN could focus on smaller tasks each time.
  \*And this method usually requires a labeled dataset (like the example above), which is quite tedious to create.
  
  What if, instead of hacking an external element to guess the alignment, we could just send the whole text and train a single neural network to both:
  
  * Somehow solve the alignment problem.
  * Use that to predict the next word.
  
  Wouldn't that be awesome? Introducing:
  
  ### Transformers
  
  They have a built-in self-attention component that scores all previous words according to their relevance for the next translated word!
  
  #### TL;DR:
  
  Transformers will automatically solve alignment while translating.
  
  
  ---
  
  