GPT-2: Why should text classification work on the last output embedding? When GPT-2 is fine-tuned for text classification (positive vs. negative), the head of the model is a linear layer that takes the LAST output embedding and outputs 2 class logits. I still can't grasp why this works. If GPT is pre-trained on predicting the next token (and it is) given a some previous sequence, then HOW can a linear layer guess the overall rating of the sequence based only on the embedding of one last output word? I've seen some explanations, but none of them were satisfying. They boiled down to the idea that the last token somehow packs the information about the entire preceding sequence. How can this be? If an embedding maps to a word, then I don't see how this embedding can contain information about the entire sequence.

For GPT models (or autoregressive in general) only last embedding is predicted based on the entire sequence, so it makes sense why last token is selected instead of any other. So the question seems to be does last embedding have information about whole sentence/text. I think that your intuition is accurate, but the question is if only linear layer is trained during fine tuning or is it trained jointly with GPT. If its trained jointly then the problem dissapears as the network learns to predict embeddings that describe more of entire sentence instead of describing single token.
