OK so I am gonna try to explain it the best I can. Say I give you a relatively long passage (512 words) and ask you to summarize it. You’re gonna do that by reading it and picking out some words that convey the gist of the passage.
In other words, your summary can be described (obviously with some oversimplification) as a weighted average of words where a few words will gain a lot of “attention” (the usage here is two-fold i.e. how we use it and also in the transformer context) and others not so much.
This is how I like to think of the [CLS] token: a weighted average of the words such that the representation of the whole sequence is captured.
Now the question arises why does [CLS] end up being this overall representation of the sequence? Well over here, the way I like to think about it is this: without fine-tuning the [CLS] token ought to be like a loose average.
However, it during the fine-tuning that the downstream task forces [CLS] weights (remember its fixed position so some transformer weights are practically dedicated to this position that can never be taken by any other token) to morph into being this optimal weighted average that the task at hand needs it to be (thanks to the good ol’ backprop).
So if you’re doing sentiment classification, the weights for adjective tokens describing the emotions end up being bigger. If you’re doing toxic tweets classfication, then abusive language words end up getting more attention and so on.
Is this all backed by research or am I telling you a nice story? Well it is the largely latter with some exceptions 
Hope it still helps!
@lewtun You are damn good at explaining stuff, what do you think?