Difference between CLS hidden state and pooled_output
Hi,

The first ouput of the TFBertModel is last_hidden_state. And I assume the CLS embdding is the first element of this object, so last_hidden_state [0]?

But then you have also the pooled_output. In the docs it is written that this comes from a linear layer on top.

1. This comes originally from the pre training. Can I imagine that this model for pre training is something like the BertForSequenceClassification?
2. Can this pooled_output be fine_tune when you fine tuning the weights of the BERT Model? I assume that this will be fixed and just the hidden states are fine tunable?
Yes so BERT (the base model without any heads on top) outputs 2 things: `last_hidden_state `and `pooler_output`.

First question:
* `last_hidden_state` contains the hidden representations for each token in each sequence of the batch. So the size is `(batch_size, seq_len, hidden_size)`. 
* `pooler_output` contains a "representation" of each sequence in the batch, and is of size `(batch_size, hidden_size)`. What it basically does is take the hidden representation of the [CLS] token of each sequence in the batch (which is a vector of size `hidden_size`), and then run that through the [`BertPooler`](https://github.com/huggingface/transformers/blob/de4d7b004a24e4bb087eb46d742ea7939bc74644/src/transformers/modeling_bert.py#L498) nn.Module. This consists of a linear layer followed by a Tanh activation function. The weights of this linear layer are already pretrained on the next sentence prediction task (note that BERT is pretrained on 2 tasks: masked language modeling and next sentence prediction). I assume that the authors of the Transformers library have taken the weights from the original TF implementation, and initialized the layer with them. In theory, they would come from [`BertForPretraining`](https://github.com/huggingface/transformers/blob/de4d7b004a24e4bb087eb46d742ea7939bc74644/src/transformers/modeling_bert.py#L862) - which is BERT with the 2 pretraining heads on top. 

Second question:
Yes you can fine-tune them, just like the hidden states, because the weights of the linear layer are updated when you perform a `loss.backward()`. 

BTW, please ask questions related to BERT/other models (which are not related to bugs) on the [forum](https://discuss.huggingface.co/), rather than posting them here.<|||||>Thank you.

Am I right that the TFBertForSequenceClassification just uses the pooled output of the main BERT model and puts it in a dropout and a dense layer with just 2 neurons?

Since this model works very well for my use cases I try to extract the encodings of the bert model and just need to feed them in a simple dense layer to reduce prediction time...

So as I understand you, this  pooling output stems from a classification head during pretraining? That is my confusiuon. Because I thought for such thing you use bert model for classficiation task with a head on top.

So if I would rebuil this sitautaion starting from just the bert model how would I intizialize my "own" BertPooler with the pretrained weights? So feedining pooled output to a dense layer with some pretrained weights like the TFBertForSequenceClassification model.

Why is actually the cls token used when it is not so good for tasks?

I would like to use other poolings or taking the average? But I think this you can do with the output of the hidden sequences. Maybe I want to feed the averaged pooled hidden sequence to the BertPooler too?
<|||||>Yes, looking at the [source code](https://github.com/huggingface/transformers/blob/aba4e22944f0c985bebdcde51d47a565dd4f551d/src/transformers/modeling_tf_bert.py#L1080) of `TFBertForSequenceClassification`, they define a dropout layer, followed by a linear layer that outputs a vector of size `config.num_labels`. 

In the [forward pass](https://github.com/huggingface/transformers/blob/aba4e22944f0c985bebdcde51d47a565dd4f551d/src/transformers/modeling_tf_bert.py#L1141), they use `outputs[1]`, meaning the output of the pooler layer (whose weights were pretrained on the next sentence classification task). This pooler layer takes the final hidden representation of the [CLS] token (this is a vector of size 768), then applies a linear layer and tanh to it, and then, we can apply the dropout layer and the linear layer which we defined in the `__init__` method. 

What is actually a bit confusing, is that in the docs they state the following about the pooler output: "This output is usually not a good summary of the semantic content of the input, you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence." (Source: https://huggingface.co/transformers/model_doc/bert.html#bertmodel)

So it's actually better to use `outputs[0]`, which are the hidden representations of all tokens, and take the average. But what actually also works well in practice is just using the hidden representation of the [CLS] token. The reason this [CLS] token is introduced is because it can be used for classification tasks. You can see the hidden representation of the [CLS] token as a representation of the whole sequence (sentence). Since `outputs[0]` is of size (batch_size, seq_len, hidden_size), and we only want the vector of the [CLS] token, we can obtain it by typing `outputs[0][:, 0, :]`. You can then apply a dropout layer, followed by a linear layer on top of that to get 2 outputs (in case you are doing binary text classification).

So, in practice this is what works well:

```
class TFBertForSequenceClassification(TFBertPreTrainedModel, TFSequenceClassificationLoss):
    def __init__(self, config, *inputs, **kwargs):
        super().__init__(config, *inputs, **kwargs)

        self.num_labels = config.num_labels
        self.bert = TFBertMainLayer(config, name="bert")
        self.dropout = tf.keras.layers.Dropout(config.hidden_dropout_prob)
        self.classifier = tf.keras.layers.Dense(
            config.num_labels, kernel_initializer=get_initializer(config.initializer_range), name="classifier"
        )

    @add_start_docstrings_to_callable(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
    @add_code_sample_docstrings(
        tokenizer_class=_TOKENIZER_FOR_DOC,
        checkpoint="bert-base-cased",
        output_type=TFSequenceClassifierOutput,
        config_class=_CONFIG_FOR_DOC,
    )
    def call(
        self,
        inputs=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        output_attentions=None,
        output_hidden_states=None,
        return_dict=None,
        labels=None,
        training=False,
    ):
        r"""
        labels (:obj:`tf.Tensor` of shape :obj:`(batch_size,)`, `optional`):
            Labels for computing the sequence classification/regression loss.
            Indices should be in :obj:`[0, ..., config.num_labels - 1]`.
            If :obj:`config.num_labels == 1` a regression loss is computed (Mean-Square loss),
            If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
        """
        return_dict = return_dict if return_dict is not None else self.bert.return_dict

        if isinstance(inputs, (tuple, list)):
            labels = inputs[9] if len(inputs) > 9 else labels
            if len(inputs) > 9:
                inputs = inputs[:9]
        elif isinstance(inputs, (dict, BatchEncoding)):
            labels = inputs.pop("labels", labels)

        outputs = self.bert(
            inputs,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
            training=training,
        )

        last_hidden_state = outputs[0]
        cls_representation = last_hidden_state[:,0,:]
        pooled_output = self.dropout(cls_representation, training=training)
        logits = self.classifier(pooled_output)
        loss = None if labels is None else self.compute_loss(labels, logits)

        if not return_dict:
            output = (logits,) + outputs[2:]
            return ((loss,) + output) if loss is not None else output

        return TFSequenceClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )
```



<|||||>I find this pooled_output very confusing because it is coming from somewhere from the "deep" and it breaks somehow the symmetry of the transformers.

That is the point of my initial question. Your above code modification is using the cls embedding. So, you could actually hjust still use the pooled_output, because it is actually the same? Ah, I forgot that the pooled_output is using the cls embeding too, but is fed to a tanh-layer then, right?


To Averaging: You mean something like GlobalAveragePooling, right? Then you have to take care about masking, right? Because for a sequence you get up to max lenght different embeddings but the actual sequence is just half along. So for averaging it is a good idea to using masking, right?
<|||||>I assume that using cls or averaging is better than pooled_output. Neverthelless I would want to try out pooled_output. So I wonder how to get pooled_output from a (finetuned) TFBertForSequenceClassification model?<|||||>I just think that it is important to use the masking of the BERT outputs for averaging?

https://discuss.huggingface.co/t/bert-output-for-padding-tokens/1550/2<|||||>Yes you're right, you should only take into account those tokens which are not padding tokens if you want to average them. I'm gonna take a look and reply later!


<|||||>Just for interest I opened an issue:

https://github.com/huggingface/transformers/issues/8148<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
