Answers.Body: <p>I took a look at the <a href="https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer_test.py#L142" rel="nofollow noreferrer">Tensor2Tensor's source code implementation</a>, and it seems like the loss function is the cross-entropy between the predicted probability matrix <span class="math-container">$\|\text{sentence length}\| \times \|\text{vocab}\|$</span> (right before taking the argmax to find the token to output), and the <span class="math-container">$\|\text{sentence length}\|$</span>-length vector of token IDs as the true label.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-10T10:21:39.500Z
Answers.Text: Note that they reshape the `out_logits` variable there, in the line that you're linking us to, i.e. `tf.reshape(out_logits, [-1, VOCAB_SIZE])`, so the cross-entropy seems to be computed between two vectors and not a matrix and a vector (but I didn't really executed that code and tried to output the shape of those tensors/variables). It may be a good idea to do it, just to confirm the exact shape.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-13T00:44:08.177Z
Answers.Text: @nbro I just ran the code, and can confirm that is the case. out_logits originally has the shape (3, 7, 1, 1, 10), which is to (BATCH_SIZE, SENTENCE_LENGTH, ??, ??, VOCAB_SIZE). The ?? are bookkeeping dimensions because the tensor2tensor model expects a 5-D tensor, but are then immediately stripped out afterwards. Then, tf.reshape(out_logits, [-1, VOCAB_SIZE]) resizes out_logits to (21, 10), which is (BATCH_SIZE * SENTENCE_LENGTH, VOCAB_SIZE). Btw, for tf.reshape, seems like adding a -1 as a dimension makes it a flexible sized dimension, so it results in a 2D matrix rather than a 1D vector.
Answers.UserDisplayName: user3667125
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-10T07:16:59.040Z
Answers.LastActivityDate: 2021-10-19T12:20:54.210Z
Answers.LastEditDate: 2021-10-19T12:20:54.210Z
Answers.UserDisplayName: user3667125
Body: <p>The paper <a href="https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="noreferrer">Attention Is All You Need</a> describes the transformer architecture that has an encoder and a decoder.</p>
  <p>However, I wasn't clear on what the cost function to minimize is for such an architecture.</p>
  <p>Consider a translation task, for example, where give an English sentence <span class="math-container">$x_{english} = [x_0, x_1, x_2, \dots, x_m]$</span>, the transformer decodes the sentence into a French sentence <span class="math-container">$x_{french}' = [x_0', x_1', \dots, x_n']$</span>. Let's say the true label is <span class="math-container">$y_{french} = [y_0, y_1, \dots, y_p]$</span>.</p>
  <p>What is the object function of the transformer? Is it the MSE between <span class="math-container">$x_{french}'$</span> and <span class="math-container">$y_{french}$</span>? And does it have any weight regularization terms?</p>
  
Tags: <neural-networks><natural-language-processing><objective-functions><transformer><attention>
Title: What is the cost function of a transformer?
UserDisplayName: user3667125
ThreadText: # What is the cost function of a transformer?
  
  (Asked by: username_0 on 2020-12-07)
  
  The paper [Attention Is All You Need](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) describes the transformer architecture that has an encoder and a decoder.
  
  However, I wasn't clear on what the cost function to minimize is for such an architecture.
  
  Consider a translation task, for example, where give an English sentence $x\_{english} = [x\_0, x\_1, x\_2, \dots, x\_m]$, the transformer decodes the sentence into a French sentence $x\_{french}' = [x\_0', x\_1', \dots, x\_n']$. Let's say the true label is $y\_{french} = [y\_0, y\_1, \dots, y\_p]$.
  
  What is the object function of the transformer? Is it the MSE between $x\_{french}'$ and $y\_{french}$? And does it have any weight regularization terms?
  
  **Tags:** neural-networks, natural-language-processing, objective-functions, transformer, attention
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-12-10. Score: 8 | Accepted answer
  
  I took a look at the [Tensor2Tensor's source code implementation](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer_test.py#L142), and it seems like the loss function is the cross-entropy between the predicted probability matrix $\|\text{sentence length}\| \times \|\text{vocab}\|$ (right before taking the argmax to find the token to output), and the $\|\text{sentence length}\|$-length vector of token IDs as the true label.
  
  > **Comments:**
  > **username_1**: Note that they reshape the `out_logits` variable there, in the line that you're linking us to, i.e. `tf.reshape(out_logits, [-1, VOCAB_SIZE])`, so the cross-entropy seems to be computed between two vectors and not a matrix and a vector (but I didn't really executed that code and tried to output the shape of those tensors/variables). It may be a good idea to do it, just to confirm the exact shape.  
  > **username_0**: @nbro I just ran the code, and can confirm that is the case. out_logits originally has the shape (3, 7, 1, 1, 10), which is to (BATCH_SIZE, SENTENCE_LENGTH, ??, ??, VOCAB_SIZE). The ?? are bookkeeping dimensions because the tensor2tensor model expects a 5-D tensor, but are then immediately stripped out afterwards. Then, tf.reshape(out_logits, [-1, VOCAB_SIZE]) resizes out_logits to (21, 10), which is (BATCH_SIZE * SENTENCE_LENGTH, VOCAB_SIZE). Btw, for tf.reshape, seems like adding a -1 as a dimension makes it a flexible sized dimension, so it results in a 2D matrix rather than a 1D vector.  
  
  ---
  
  