I’ve read that statement a number of times, indeed, though I have never encountered any issues with it - although I typically use my own configuration. For a sentence classification task I have always followed the results from the original BERT layer: concatenating the hidden states of the last four layers, taking the output from [CLS], and adding a linear layer, activation, and final linear layer on top.
My guess is, as always, that it depends on your specific use case, dataset, optimizer.
Perhaps the warning is that the CLS item is not a good semantic representation. So if you want to use that final hidden state as “semantic features” in another system , you may want to opt for the mean over the tokens or something like that. But if you are going straight into the classification problem after the transformer layers, you may can resort to [CLS]? Not sure.