Answers.Body: <p>A linear layer followed by Softmax is applied to the transformer output to get the probabilities for all tokens in the vocabulary, and then the next token/word is chosen based on these probabilities.</p>
  <p>Refer to the Fig 1. of the <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Transformer paper</a> by Vaswani et al. (2017)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-27T19:58:56.340Z
Answers.LastActivityDate: 2024-08-29T03:51:57.157Z
Answers.LastEditDate: 2024-08-29T03:51:57.157Z
Answers.UserDisplayName: Kulin Patel
Body: <p>I am quite new to the topic of transformers. In a lecture of mine, we defined transformers as sequence to sequence mappings, i.e. a transformer <span class="math-container">$\mathcal{T}$</span> takes <span class="math-container">$X \in \mathbb{R}^{n \times d}$</span> and maps it to <span class="math-container">$\mathcal{T}(X) \in \mathbb{R}^{n \times d}$</span>. Of course, under the hood there are transformer blocks, attention layers, etc.</p>
  <p>It is my understanding that <span class="math-container">$n$</span> represents the size of the context window and <span class="math-container">$d$</span> is the size of the word embedding. For example, suppose we have the sentence 'Gras is' and we want to use an already trained transformer. I understand that something like this is happening now:
  <span class="math-container">$$
  \text{Gras is} \overset{embedding}{\longrightarrow}
  \begin{pmatrix}
  0.6 \\
  0.8 \\
  -0.2
  \end{pmatrix}
  \begin{pmatrix}
  0.78 \\
  -0.03 \\
  -1.03
  \end{pmatrix}
  \approx
  \begin{pmatrix}
  0.6 &amp; 0.78\\
  0.8 &amp; -0.03\\
  -0.2 &amp; -1.03
  \end{pmatrix}
  \overset{\mathcal{T}}{\longrightarrow}
  \underbrace{\begin{pmatrix}
  0.89 &amp; -0.7\\
  0.2 &amp; 1.23\\
  -0.1 &amp; 0.24
  \end{pmatrix}}_{=: A}
  \overset{???}{\longrightarrow}
  \text{Next word}.
  $$</span>
  So we tokenise the words 'Gras' and 'is', embed each word in <span class="math-container">$\mathbb{R}^3$</span>, apply the transformer and end up with a new <span class="math-container">$2 \times 3$</span> matrix. Hence the context window is <span class="math-container">$n = 2$</span> and the embedding dimension is <span class="math-container">$d = 3$</span>.</p>
  <p><strong>Question:</strong> How is the next word chosen?</p>
  <p>I know that the transformer now uses the output matrix to sample the next word, but how does the work concretely. Suppose we have an alphabet of length <span class="math-container">$500$</span> and each word gets a vector embedding, leaving us with vectors <span class="math-container">$v_1, \dots, v_{500} \in \mathbb{R}^3$</span>.</p>
  <p>Based on what information from the matrix <span class="math-container">$A$</span> do I choose the next vector <span class="math-container">$v_i$</span> for <span class="math-container">$i=1,\dots,500$</span>? E.g. one could look which vector <span class="math-container">$v_i$</span> is closest (in Euclidean distance) from e.g. the second column from <span class="math-container">$A$</span>. But how is it really done?</p>
  <p>I know that my understanding and depiction of a transformer is <strong>very</strong> basic, but I would like to understand how the next token is generated. I would be happy if someone could shed some light onto this. And feel free to comment on my understanding of a transformer in general if there is something completely wrong.</p>
  <p>Thanks in advance!</p>
  
Tags: <natural-language-processing><transformer><word-embedding>
Title: How does a transformer choose the next word concretely?
UserDisplayName: julian2000P
ThreadText: # How does a transformer choose the next word concretely?
  
  (Asked by: username_0 on 2024-08-24)
  
  I am quite new to the topic of transformers. In a lecture of mine, we defined transformers as sequence to sequence mappings, i.e. a transformer $\mathcal{T}$ takes $X \in \mathbb{R}^{n \times d}$ and maps it to $\mathcal{T}(X) \in \mathbb{R}^{n \times d}$. Of course, under the hood there are transformer blocks, attention layers, etc.
  
  It is my understanding that $n$ represents the size of the context window and $d$ is the size of the word embedding. For example, suppose we have the sentence 'Gras is' and we want to use an already trained transformer. I understand that something like this is happening now:
  $$
  \text{Gras is} \overset{embedding}{\longrightarrow}
  \begin{pmatrix}
  0.6 \\
  0.8 \\
  -0.2
  \end{pmatrix}
  \begin{pmatrix}
  0.78 \\
  -0.03 \\
  -1.03
  \end{pmatrix}
  \approx
  \begin{pmatrix}
  0.6 & 0.78\\
  0.8 & -0.03\\
  -0.2 & -1.03
  \end{pmatrix}
  \overset{\mathcal{T}}{\longrightarrow}
  \underbrace{\begin{pmatrix}
  0.89 & -0.7\\
  0.2 & 1.23\\
  -0.1 & 0.24
  \end{pmatrix}}\_{=: A}
  \overset{???}{\longrightarrow}
  \text{Next word}.
  $$
  So we tokenise the words 'Gras' and 'is', embed each word in $\mathbb{R}^3$, apply the transformer and end up with a new $2 \times 3$ matrix. Hence the context window is $n = 2$ and the embedding dimension is $d = 3$.
  
  **Question:** How is the next word chosen?
  
  I know that the transformer now uses the output matrix to sample the next word, but how does the work concretely. Suppose we have an alphabet of length $500$ and each word gets a vector embedding, leaving us with vectors $v\_1, \dots, v\_{500} \in \mathbb{R}^3$.
  
  Based on what information from the matrix $A$ do I choose the next vector $v\_i$ for $i=1,\dots,500$? E.g. one could look which vector $v\_i$ is closest (in Euclidean distance) from e.g. the second column from $A$. But how is it really done?
  
  I know that my understanding and depiction of a transformer is **very** basic, but I would like to understand how the next token is generated. I would be happy if someone could shed some light onto this. And feel free to comment on my understanding of a transformer in general if there is something completely wrong.
  
  Thanks in advance!
  
  **Tags:** natural-language-processing, transformer, word-embedding
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-08-27. Score: 1 | Accepted answer
  
  A linear layer followed by Softmax is applied to the transformer output to get the probabilities for all tokens in the vocabulary, and then the next token/word is chosen based on these probabilities.
  
  Refer to the Fig 1. of the [Transformer paper](https://arxiv.org/abs/1706.03762) by Vaswani et al. (2017)
  
  
  ---
  
  