Why the output of DistilBertModel is inconsistent with BertModel?!
[The output of DistilBertModel](https://github.com/huggingface/transformers/blob/be916cb3fb4579e278ceeaec11a6524662797d7f/transformers/modeling_distilbert.py#L468) does not contain [pooled_output as available in BERT model](https://github.com/huggingface/transformers/blob/be916cb3fb4579e278ceeaec11a6524662797d7f/transformers/modeling_bert.py#L632).
I'm going to replace BERT with DistilBert in a classification task, if pooled_output is not a proper way in DistillBertModel? -Currently, I'm using pooled_output of BERT in my experiments.
Hello @amirj,

The "pooled_output" is the hidden state of the `[CLS]`. It is this hidden state that is used for classification tasks for instance (see DistilBertForSequenceClassification). So you could retrieve it by filtering out `hidden_states `.

The reason why there is no linear transformation in the pooler of DistilBERT is because I'm removing the next sentence prediction (see RoBERTa which also do the same). However, this linear transformation is still in `DistilBertForSequenceClassification` so that the classification heads for DistilBERT and BERT have the same number of parameters.

I hope it answers your question.
Victor<|||||>Hello @VictorSanh,
DistilBertForSequenceClassification is the rescue.
Thanks.
Amir<|||||>def forward(self, input_ids, attention_mask, labels=None):
    output = self.bert(input_ids, attention_mask = attention_mask)
    output = self.classifier(output.hidden_states) 
    output = torch.sigmoid(output)    
    loss = 0
    if labels is not None:
        loss = self.criterion(output, labels)
    return loss, output

for BERT model self.classifier will take output.pooler_output as its input
But for DistilBERT this doesnt happen.
I used "DistilBertForSequenceClassification " and replaced pooler_output with hidden_states, still it doesnt work

Error I get is: linear(): argument 'input' (position 1) must be Tensor, not NoneType