default output of BertModel.from_pretrained('bert-base-uncased')
By default `output = BertModel.from_pretrained('bert-base-uncased')` is a 2-tuple where `output[0]` is the hidden states of the last layer, but how is `output[1]` computed? It doesn't seem to be average of the last layer hidden states vectors over multiple tokens. I am trying to leverage output as sentence embedding, not sure if I should use `output[1]`. Thank you!
See https://huggingface.co/transformers/v1.2.0/_modules/pytorch_transformers/modeling_bert.html#BertModel

I think this is your output[1]:
"Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during Bert pretraining. This output is usually **not** a good summary of the semantic content of the input, you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence."<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
