Yeah I think you see it as CLS as just any other token but for me the fact that it has its own dedicated position and that it is there in every single pretraining/finetuning step at that position already makes it uniquely positioned to carry sentence-level semantics.
Would be nice to see others’ take  I am the farthest thing from authority on this subject.