Answers.Body: <p>These answers are based on my personal understanding of Bert from both the <a href="https://arxiv.org/abs/1810.04805" rel="nofollow noreferrer">paper</a> and <a href="https://github.com/google-research/bert" rel="nofollow noreferrer">official_implementation</a>, hope it will help:</p>
  
  <pre><code>What do they mean by "maximum scoring span is used as the prediction"?
  </code></pre>
  
  <p>As you know in SQuAD the input sequence is divided to 2 parts: Question and Document (from which we extract the answer if possible). </p>
  
  <p>Sometimes the input length exceeds the <code>max_seq_length</code> parameter, in this case the document is truncated to as many parts as needed and we end up having more than one input for the same question/document. Mention that the question is replicated in all the resulting inputs (see <a href="https://github.com/google-research/bert/blob/master/run_squad.py" rel="nofollow noreferrer">line_350</a> for details). </p>
  
  <p>So in such cases, in order to determine the <strong>predicted span</strong> among all the generated-inputs we use the <strong>maximum scoring</strong> <em>i.e</em> <code>(max_start + max_end) / 2 = ( max(Softmax(S*Ti)) + max(Softmax(E*Ti)) ) / 2</code> of all the inputs related to the same question.</p>
  
  <pre><code>"The training objective is the loglikelihood of the correct start and end positions"
  </code></pre>
  
  <p>The loss is the average of the start_position loss <code>start_loss</code> and the end_postition loss <code>end_loss</code>. Each loss is computed in the same way: after applying the Softmax to the final output (i.e <code>S*Ti</code> or <code>E*Ti</code>) we use the real start/end postitions to compute the loss (see code below).</p>
  
  <p>From <a href="https://github.com/google-research/bert/blob/master/run_squad.py" rel="nofollow noreferrer">run_squad.py</a> in Bert_repo:</p>
  
  <pre><code> def compute_loss(logits, positions):
          one_hot_positions = tf.one_hot(positions, depth=seq_length, dtype=tf.float32)
          log_probs = tf.nn.log_softmax(logits, axis=-1)
          loss = -tf.reduce_mean(tf.reduce_sum(one_hot_positions * log_probs, axis=-1))
          return loss
  
   start_positions = features["start_positions"]
   end_positions = features["end_positions"]
  
   start_loss = compute_loss(start_logits, start_positions)
   end_loss = compute_loss(end_logits, end_positions)
  
   total_loss = (start_loss + end_loss) / 2.0
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-22T19:58:58.523Z
Answers.LastActivityDate: 2019-04-22T21:33:07.277Z
Answers.LastEditDate: 2019-04-22T21:33:07.277Z
Answers.UserDisplayName: HLeb
Body: <p>In the <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="nofollow noreferrer">BERT paper</a>, section 4.2 covers the SQuAD training. </p>
  
  <p>From my understanding, there are two extra parameters trained, they are two vectors with the same dimension as the hidden size, so the same dimensions as the contextualized embeddings in BERT. They are S (for start) and E (for End). For each, a softmax is taken with S and each of the final contextualized embeddings to get a score for the correct Start position. And the same thing is done for E and the correct end position.</p>
  
  <p>I get up to this part. But I am having trouble figuring out how the did the labeling and final loss calculations, which is described in this paragraph</p>
  
  <blockquote>
    <p>and the maximum scoring span is used as the prediction. The training objective is the loglikelihood of the correct start and end positions.</p>
  </blockquote>
  
  <p>What do they mean by "maximum scoring span is used as the prediction"?</p>
  
  <p>Furthermore, how does that play into </p>
  
  <blockquote>
    <p>The training objective is the loglikelihood of the correct start and end positions</p>
  </blockquote>
  
  <p>From this source: <a href="https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/" rel="nofollow noreferrer">https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/</a></p>
  
  <p>It says the log-likelihood is only applied to the correct classes. So, we are only calculating the softmax for the correct positions only, not any of the incorrect positions.</p>
  
  <p>If this interpretation is correct, then the loss will be</p>
  
  <pre><code>Loss = -Log( Softmax(S*T(predictedStart) / Sum(S*Ti) ) -Log( Softmax(E*T(predictedEnd) / Sum(S*Ti) )
  </code></pre>
  
Tags: <natural-language-processing><objective-functions><papers><bert>
Title: Understanding how the loss was calculated for the SQuAD task in BERT paper
UserDisplayName: SantoshGupta7
ThreadText: # Understanding how the loss was calculated for the SQuAD task in BERT paper
  
  (Asked by: username_0 on 2019-04-20)
  
  In the [BERT paper](https://arxiv.org/pdf/1810.04805.pdf), section 4.2 covers the SQuAD training.
  
  From my understanding, there are two extra parameters trained, they are two vectors with the same dimension as the hidden size, so the same dimensions as the contextualized embeddings in BERT. They are S (for start) and E (for End). For each, a softmax is taken with S and each of the final contextualized embeddings to get a score for the correct Start position. And the same thing is done for E and the correct end position.
  
  I get up to this part. But I am having trouble figuring out how the did the labeling and final loss calculations, which is described in this paragraph
  
  > and the maximum scoring span is used as the prediction. The training objective is the loglikelihood of the correct start and end positions.
  
  What do they mean by "maximum scoring span is used as the prediction"?
  
  Furthermore, how does that play into
  
  > The training objective is the loglikelihood of the correct start and end positions
  
  From this source: <https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/>
  
  It says the log-likelihood is only applied to the correct classes. So, we are only calculating the softmax for the correct positions only, not any of the incorrect positions.
  
  If this interpretation is correct, then the loss will be
  
  ```
  Loss = -Log( Softmax(S*T(predictedStart) / Sum(S*Ti) ) -Log( Softmax(E*T(predictedEnd) / Sum(S*Ti) )
  
  ```
  
  **Tags:** natural-language-processing, objective-functions, papers, bert
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-04-22. Score: 2 | Accepted answer
  
  These answers are based on my personal understanding of Bert from both the [paper](https://arxiv.org/abs/1810.04805) and [official\_implementation](https://github.com/google-research/bert), hope it will help:
  
  ```
  What do they mean by "maximum scoring span is used as the prediction"?
  
  ```
  
  As you know in SQuAD the input sequence is divided to 2 parts: Question and Document (from which we extract the answer if possible).
  
  Sometimes the input length exceeds the `max_seq_length` parameter, in this case the document is truncated to as many parts as needed and we end up having more than one input for the same question/document. Mention that the question is replicated in all the resulting inputs (see [line\_350](https://github.com/google-research/bert/blob/master/run_squad.py) for details).
  
  So in such cases, in order to determine the **predicted span** among all the generated-inputs we use the **maximum scoring** *i.e* `(max_start + max_end) / 2 = ( max(Softmax(S*Ti)) + max(Softmax(E*Ti)) ) / 2` of all the inputs related to the same question.
  
  ```
  "The training objective is the loglikelihood of the correct start and end positions"
  
  ```
  
  The loss is the average of the start\_position loss `start_loss` and the end\_postition loss `end_loss`. Each loss is computed in the same way: after applying the Softmax to the final output (i.e `S*Ti` or `E*Ti`) we use the real start/end postitions to compute the loss (see code below).
  
  From [run\_squad.py](https://github.com/google-research/bert/blob/master/run_squad.py) in Bert\_repo:
  
  ```
   def compute_loss(logits, positions):
          one_hot_positions = tf.one_hot(positions, depth=seq_length, dtype=tf.float32)
          log_probs = tf.nn.log_softmax(logits, axis=-1)
          loss = -tf.reduce_mean(tf.reduce_sum(one_hot_positions * log_probs, axis=-1))
          return loss
  
   start_positions = features["start_positions"]
   end_positions = features["end_positions"]
  
   start_loss = compute_loss(start_logits, start_positions)
   end_loss = compute_loss(end_logits, end_positions)
  
   total_loss = (start_loss + end_loss) / 2.0
  
  ```
  
  
  ---
  
  