In the upcoming book “Natural Language Processing with Transformers”, it’s teaching us how to do a classification task on sentences by using Transformers as Feature Extractors. We process the sentences through a transformer to get the hidden state.
To train a classifier, we take the token embedding for just the first token, namely the “[CLS]” and ignore the rest of the sentence. The book says that it’s common practice to do that.
It doesn’t make much sense to me to ignore the rest of the embeddings. Shouldn’t they be averaged or something?
The only reasoning I can think of is that the attention layers of the encoder make the CLS token absorb the meaningful context?
Thank you! The book is awesome by the way, highly recommended!