BertForSequenceClassification and DistilBertForSequenceClassification use pooler output in different ways
Hi,
The `BertForSequenceClassification` includes a forward pass of the BertModel, and it takes the second element (index 1) from its output before moving forward, as shown [here ](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py#L1378 )
This is the return of `BertModel`
```
    return BaseModelOutputWithPoolingAndCrossAttentions(
            last_hidden_state=sequence_output,
            pooler_output=pooled_output,
            hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,
            cross_attentions=encoder_outputs.cross_attentions,
        )
```
hence `output[1]` is taking the pooler_output.

However, in `DistilBertForSequenceClassification`, it takes the first element (index 0) of the `DistilBertModel`'s forward pass, i.e. `distilbert_output[0]`, as shown [here](https://github.com/huggingface/transformers/blob/master/src/transformers/models/distilbert/modeling_distilbert.py#L625)

This is the last hidden state for all tokens.
Why is there this discrepancy between the two models? The behaviour of `DistilBertForSequenceClassification` makes more intuitive sense to me.
Why is `BertForSequenceClassification` using only the pooler_output from `BertModel` (i.e. the hidden state of the first token)? Why are all other hidden states of other tokens not needed here, but needed in the distilled version?

Thanks for the help!
    
BERT and DistilBERT are different models. DistilBERT isn't simply a BERT model with fewer layers, but a BERT model without the pooling layer as you have seen, and with no token type embeddings.

We try to stay as close to the original implementations as possible, hence why BERT is done this way, and why DistilBERT was done differently. I invite you to read the paper or study the original BERT codebase to see how it was done, it should be very similar (or the same) as it is done here.<|||||>Thanks a lot for clarifying @LysandreJik I'll close the issue then!