Well, intuitively you would think that taking the average over all the last hidden states of all tokens captures a bit better the essence of the sentence than just taking the CLS token. The meaning of tokens after pretraining are usable (just take average and you have a relatively decent semantic representation), but CLS is specific to the NSP task.