Answers.Body: <p>This seems to be inherited from the original Google implementation, which also uses 2 outputs (<a href="https://github.com/google-research/bert/blob/master/run_pretraining.py#L293" rel="nofollow noreferrer">https://github.com/google-research/bert/blob/master/run_pretraining.py#L293</a>). I can see two possible reasons that the original implementation uses 2 outputs:</p>
  <ol>
  <li><p>They are using the cross entropy loss, which typically works with log probabilities. To get probabilities they use softmax activation, which requires an output for each class. It is possible, of course, to compute cross entropy from sigmoid activations (which would correspond to a 1-output architecture), but <a href="https://stackoverflow.com/questions/59058867/sigmoid-output-can-it-be-interpreted-as-probability">there seems to be some confusion</a> as to whether the output of the sigmoid function should be used as a probability.</p>
  </li>
  <li><p>Using 2 outputs can simplify the computation of the binary cross entropy loss, which, in typical Google fashion, is computed using low-level tensorflow ops rather than with <code>tf.nn.softmax_cross_entropy_with_logits</code>. Specifically,</p>
  </li>
  </ol>
  <pre><code>-tf.reduce_sum(one_hot_labels * log_probs, axis=-1)
  </code></pre>
  <p>where <code>one_hot_labels</code> and <code>log_probs</code> are <span class="math-container">$\mathbb{R}^{N \times 2}$</span>, is much easier to read than</p>
  <pre><code>-tf.reduce_sum(binary_labels * tf.math.log(probs) + (1 - binary_labels) + tf.math.log(1 - probs))
  </code></pre>
  <p>where <code>binary_labels</code> and <code>probs</code> are <span class="math-container">$\mathbb{R}^N$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-23T14:38:55.440Z
Answers.LastActivityDate: 2020-06-23T14:38:55.440Z
Answers.UserDisplayName: primussucks
Body: <p>Here's the code in question. </p>
  
  <p><a href="https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491" rel="nofollow noreferrer">https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491</a></p>
  
  <pre><code>class BertOnlyNSPHead(nn.Module):
      def __init__(self, config):
          super().__init__()
          self.seq_relationship = nn.Linear(config.hidden_size, 2)
  
      def forward(self, pooled_output):
          seq_relationship_score = self.seq_relationship(pooled_output)
          return seq_relationship_score
  </code></pre>
  
  <p>I think it was just ranking how likely one sentence would follow another? Wouldn't it be one score?</p>
  
Tags: <machine-learning><natural-language-processing><transformer><bert>
Title: Why does the BERT NSP head linear layer have two outputs?
UserDisplayName: SantoshGupta7
ThreadText: # Why does the BERT NSP head linear layer have two outputs?
  
  (Asked by: username_0 on 2020-05-30)
  
  Here's the code in question.
  
  <https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491>
  
  ```
  class BertOnlyNSPHead(nn.Module):
      def __init__(self, config):
          super().__init__()
          self.seq_relationship = nn.Linear(config.hidden_size, 2)
  
      def forward(self, pooled_output):
          seq_relationship_score = self.seq_relationship(pooled_output)
          return seq_relationship_score
  
  ```
  
  I think it was just ranking how likely one sentence would follow another? Wouldn't it be one score?
  
  **Tags:** machine-learning, natural-language-processing, transformer, bert
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-23. Score: 1 | Accepted answer
  
  This seems to be inherited from the original Google implementation, which also uses 2 outputs (<https://github.com/google-research/bert/blob/master/run_pretraining.py#L293>). I can see two possible reasons that the original implementation uses 2 outputs:
  
  1. They are using the cross entropy loss, which typically works with log probabilities. To get probabilities they use softmax activation, which requires an output for each class. It is possible, of course, to compute cross entropy from sigmoid activations (which would correspond to a 1-output architecture), but [there seems to be some confusion](https://stackoverflow.com/questions/59058867/sigmoid-output-can-it-be-interpreted-as-probability) as to whether the output of the sigmoid function should be used as a probability.
  2. Using 2 outputs can simplify the computation of the binary cross entropy loss, which, in typical Google fashion, is computed using low-level tensorflow ops rather than with `tf.nn.softmax_cross_entropy_with_logits`. Specifically,
  
  ```
  -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)
  
  ```
  
  where `one_hot_labels` and `log_probs` are $\mathbb{R}^{N \times 2}$, is much easier to read than
  
  ```
  -tf.reduce_sum(binary_labels * tf.math.log(probs) + (1 - binary_labels) + tf.math.log(1 - probs))
  
  ```
  
  where `binary_labels` and `probs` are $\mathbb{R}^N$.
  
  
  ---
  
  