why take the first hidden state for sequence classification (DistilBertForSequenceClassification)
In the last few layers of sequence classification [here][1], the first hidden state of the sequence length of the transformer output to be used for classification. 

    hidden_state = distilbert_output[0]  # (bs, seq_len, dim) <-- transformer output
    pooled_output = hidden_state[:, 0]  # (bs, dim)           <-- first hidden state
    pooled_output = self.pre_classifier(pooled_output)  # (bs, dim)
    pooled_output = nn.ReLU()(pooled_output)  # (bs, dim)
    pooled_output = self.dropout(pooled_output)  # (bs, dim)
    logits = self.classifier(pooled_output)  # (bs, dim)

Is there any benefit to taking the first hidden state over the last, average, or even the use of a Flatten layer instead?

I've also asked this question on [Stack Overflow](https://stackoverflow.com/questions/60087613/why-take-the-first-hidden-state-for-sequence-classification-distilbertforsequen)


  [1]: https://github.com/huggingface/transformers/blob/33d3072e1c54bcd235447b98c6dea1b4cb71234c/src/transformers/modeling_distilbert.py#L634
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
