Answers.Body: <p>It's about <em>self-attention</em>, a mechanism that targets <em>parallelism</em> among other goals (see <a href="https://arxiv.org/pdf/1706.03762.pdf#section.4" rel="nofollow noreferrer">1706.03762.pdf - Why Self-Attention</a>).</p>
  <p>From <a href="https://blogs.nvidia.com/blog/2022/03/25/what-is-a-transformer-model/" rel="nofollow noreferrer">What Is a Transformer Model? | NVIDIA Blogs</a>:</p>
  <blockquote>
  <p>How Transformers Got Their Name</p>
  <p>Attention is so key to transformers the Google researchers almost used the term as the name for their 2017 model. Almost.</p>
  <p>“Attention Net didn’t sound very exciting,” said Vaswani, who started working with neural nets in 2011.</p>
  <p>Jakob Uszkoreit, a senior software engineer on the team, came up with the name Transformer.</p>
  <p>“I argued we were transforming representations, but that was just playing semantics,” Vaswani said.</p>
  </blockquote>
  <p>You'll see the same sentiment in the first paragraph of <a href="https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)" rel="nofollow noreferrer">Transformer (machine learning model)</a>.</p>
  <p>This is not to say that self-attention is the only the kind of attention mechanism employed by a Transformer model; see <a href="https://arxiv.org/pdf/1706.03762.pdf#subsubsection.3.2.3" rel="nofollow noreferrer">1706.03762.pdf - 3.2.3 Applications of Attention in our Model</a>. It's also not to say that adding an attention mechanism to your model is to make it a &quot;Transformer&quot; model; the innovation of this particular model was to go &quot;all out&quot; on attention mechanisms (you'd also have to get rid of recurrent and convolutional components).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T20:07:10.863Z
Answers.LastActivityDate: 2024-04-19T14:34:40.203Z
Answers.LastEditDate: 2024-04-19T14:34:40.203Z
Answers.UserDisplayName: davidvandebunte
Answers.Body: <p>There is not one answer to this question, but one could argue that transformers heavily rely on</p>
  <ul>
  <li>transforming each input into latent subspaces of queries, keys and values in order to generate attention score</li>
  <li>a pool of transformations of the attention vectors (multi-head) according to which models can capture richer interpretations as different sections of the input embedding can attend different per-head subspaces that link back to each input</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-27T12:03:06.347Z
Answers.LastActivityDate: 2021-05-27T12:03:06.347Z
Answers.UserDisplayName: hH1sG0n3
Body: <p>Transformers are modified heavily in recent research. But what exactly makes a transformer a transformer? What is the core part of a transformer? Is it the <em>self-attention</em>, the <em>parallelism</em>, or something else?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-27T08:58:42.463Z
Comments.Text: When you say "Transformers are modified heavily in recent research", which research are you talking about that "modified heavily" the original transformer? In any case, [here](https://ai.stackexchange.com/q/24643/2444) and [here](https://ai.stackexchange.com/q/27326/2444) are 2 related questions.
Comments.UserDisplayName: nbro
Tags: <deep-learning><definitions><transformer>
Title: What makes a transformer a transformer?
UserDisplayName: AB Saravanan
ThreadText: # What makes a transformer a transformer?
  
  (Asked by: username_0 on 2021-05-27)
  
  Transformers are modified heavily in recent research. But what exactly makes a transformer a transformer? What is the core part of a transformer? Is it the *self-attention*, the *parallelism*, or something else?
  
  **Tags:** deep-learning, definitions, transformer
  > **Comments:**
  > **username_1**: When you say "Transformers are modified heavily in recent research", which research are you talking about that "modified heavily" the original transformer? In any case, [here](https://ai.stackexchange.com/q/24643/2444) and [here](https://ai.stackexchange.com/q/27326/2444) are 2 related questions.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-06-12. Score: 4
  
  It's about *self-attention*, a mechanism that targets *parallelism* among other goals (see [1706.03762.pdf - Why Self-Attention](https://arxiv.org/pdf/1706.03762.pdf#section.4)).
  
  From [What Is a Transformer Model? | NVIDIA Blogs](https://blogs.nvidia.com/blog/2022/03/25/what-is-a-transformer-model/):
  
  > How Transformers Got Their Name
  >
  > Attention is so key to transformers the Google researchers almost used the term as the name for their 2017 model. Almost.
  >
  > “Attention Net didn’t sound very exciting,” said Vaswani, who started working with neural nets in 2011.
  >
  > Jakob Uszkoreit, a senior software engineer on the team, came up with the name Transformer.
  >
  > “I argued we were transforming representations, but that was just playing semantics,” Vaswani said.
  
  You'll see the same sentiment in the first paragraph of [Transformer (machine learning model)](https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)).
  
  This is not to say that self-attention is the only the kind of attention mechanism employed by a Transformer model; see [1706.03762.pdf - 3.2.3 Applications of Attention in our Model](https://arxiv.org/pdf/1706.03762.pdf#subsubsection.3.2.3). It's also not to say that adding an attention mechanism to your model is to make it a "Transformer" model; the innovation of this particular model was to go "all out" on attention mechanisms (you'd also have to get rid of recurrent and convolutional components).
  
  
  ---
  
  ### Answer by username_3 on 2021-05-27. Score: 1
  
  There is not one answer to this question, but one could argue that transformers heavily rely on
  
  * transforming each input into latent subspaces of queries, keys and values in order to generate attention score
  * a pool of transformations of the attention vectors (multi-head) according to which models can capture richer interpretations as different sections of the input embedding can attend different per-head subspaces that link back to each input
  
  
  ---
  
  