For sequence classification, is this model using the wrong token?
According to the BERT paper, we want to use the weights for the `[CLS]` token, which – as far as I understand – would be the first hidden output here, not the last?

i.e. shouldn't this be `encoded_layers[0]` below?

https://github.com/huggingface/pytorch-pretrained-BERT/blob/f7c9dc8c998395d2ad9edbf0fd6fa072f03cc667/pytorch_pretrained_bert/modeling.py#L715-L719

@thomwolf Let me know if I'm missing something and if you're concerned about this breaking anything else and I can PR.
Never mind, I think I got confused. Is it that the pooler already takes care of that and encoded_layers represents the pooler output for each attention layer?<|||||>Hi Catalin, the content of the outputs (`encoded_layers` and `pooled_output`) is detailed in the readme [here](https://github.com/huggingface/pytorch-pretrained-BERT#1-bertmodel) and in the model docstring. `encoded_layers` contains the encoded-hidden-states at the end of each attention block (i.e. 12 full sequences for BERT-base, 24 for BERT-large), each encoded-hidden-state is a torch.FloatTensor of size [batch_size, sequence_length, hidden_size].

The Pooler take care of extracting the hidden state of the first token from the encoded hidden state of the full sequence (see [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/f7c9dc8c998395d2ad9edbf0fd6fa072f03cc667/pytorch_pretrained_bert/modeling.py#L413))