<p>This seems to be inherited from the original Google implementation, which also uses 2 outputs (<a href="https://github.com/google-research/bert/blob/master/run_pretraining.py#L293" rel="nofollow noreferrer">https://github.com/google-research/bert/blob/master/run_pretraining.py#L293</a>). I can see two possible reasons that the original implementation uses 2 outputs:</p>
<ol>
<li><p>They are using the cross entropy loss, which typically works with log probabilities. To get probabilities they use softmax activation, which requires an output for each class. It is possible, of course, to compute cross entropy from sigmoid activations (which would correspond to a 1-output architecture), but <a href="https://stackoverflow.com/questions/59058867/sigmoid-output-can-it-be-interpreted-as-probability">there seems to be some confusion</a> as to whether the output of the sigmoid function should be used as a probability.</p>
</li>
<li><p>Using 2 outputs can simplify the computation of the binary cross entropy loss, which, in typical Google fashion, is computed using low-level tensorflow ops rather than with <code>tf.nn.softmax_cross_entropy_with_logits</code>. Specifically,</p>
</li>
</ol>
<pre><code>-tf.reduce_sum(one_hot_labels * log_probs, axis=-1)
</code></pre>
<p>where <code>one_hot_labels</code> and <code>log_probs</code> are <span class="math-container">$\mathbb{R}^{N \times 2}$</span>, is much easier to read than</p>
<pre><code>-tf.reduce_sum(binary_labels * tf.math.log(probs) + (1 - binary_labels) + tf.math.log(1 - probs))
</code></pre>
<p>where <code>binary_labels</code> and <code>probs</code> are <span class="math-container">$\mathbb{R}^N$</span>.</p>

CC BY-SA 4.0
2020-06-23T14:38:55.440Z
2020-06-23T14:38:55.440Z
primussucks
<p>Here's the code in question. </p>

<p><a href="https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491" rel="nofollow noreferrer">https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491</a></p>

<pre><code>class BertOnlyNSPHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.seq_relationship = nn.Linear(config.hidden_size, 2)

    def forward(self, pooled_output):
        seq_relationship_score = self.seq_relationship(pooled_output)
        return seq_relationship_score
</code></pre>

<p>I think it was just ranking how likely one sentence would follow another? Wouldn't it be one score?</p>

<machine-learning><natural-language-processing><transformer><bert>
Why does the BERT NSP head linear layer have two outputs?
SantoshGupta7
# Why does the BERT NSP head linear layer have two outputs?

(Asked by: username_0 on 2020-05-30)

Here's the code in question. 
https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491
class BertOnlyNSPHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.seq_relationship = nn.Linear(config.hidden_size, 2)

    def forward(self, pooled_output):
        seq_relationship_score = self.seq_relationship(pooled_output)
        return seq_relationship_score

I think it was just ranking how likely one sentence would follow another? Wouldn't it be one score?

**Tags:** machine-learning, natural-language-processing, transformer, bert


---

## 1 answers

### Answer by username_1 on 2020-06-23. Score: 1 | Accepted answer

This seems to be inherited from the original Google implementation, which also uses 2 outputs (https://github.com/google-research/bert/blob/master/run_pretraining.py#L293). I can see two possible reasons that the original implementation uses 2 outputs:

They are using the cross entropy loss, which typically works with log probabilities. To get probabilities they use softmax activation, which requires an output for each class. It is possible, of course, to compute cross entropy from sigmoid activations (which would correspond to a 1-output architecture), but there seems to be some confusion as to whether the output of the sigmoid function should be used as a probability.

Using 2 outputs can simplify the computation of the binary cross entropy loss, which, in typical Google fashion, is computed using low-level tensorflow ops rather than with tf.nn.softmax_cross_entropy_with_logits. Specifically,


-tf.reduce_sum(one_hot_labels * log_probs, axis=-1)

where one_hot_labels and log_probs are $\mathbb{R}^{N \times 2}$, is much easier to read than
-tf.reduce_sum(binary_labels * tf.math.log(probs) + (1 - binary_labels) + tf.math.log(1 - probs))

where binary_labels and probs are $\mathbb{R}^N$.


---

