SequenceSummary / quenstion regarding summary types
in the class SequenceSummary(nn.Module) which is a part of {BERT, XLNet}ForSequenceClassification:

https://github.com/huggingface/pytorch-transformers/blob/32e1332acf6fd1ad372b81c296d43be441d3b0b1/pytorch_transformers/modeling_utils.py#L643-L644

in case of XLNet we can see that the last token will be taken, for BERT it is otherwise the first one.
https://github.com/huggingface/pytorch-transformers/blob/32e1332acf6fd1ad372b81c296d43be441d3b0b1/pytorch_transformers/modeling_utils.py#L692

Why only the first one or the last one? Why not to apply the max or average pooling over all token from the output? 

Thanks a lot!
That's how it's done in the respective original implementations.<|||||>it is very interesting. All other tokens will not be considered. 

Do you know whether other architectures has been tried out? Initially, when I had taken the transformer-model I created another output architecture. It is very interesting, whether this "end architecture" affects the performance significantly

Thanks a lot!<|||||>I found this project doc [1], there are four variants of the end architecture, their performance is almost equal. 

I will be glad if you can provide other papers regarding this issue. 

[1] - http://web.stanford.edu/class/cs224n/reports/custom/15785631.pdf<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
