What you mention is true for all tokens, not only for CLS. The value of CLS is influenced by other tokens, just like other tokens are influenced by their context (attention).
But that is not what a weighted average is. CLS still has its own embedding token and starts from that. It is not only the result of operations on the other tokens.