I suspect you might be mixing up notions of BERT pre-training and BERT fine-tuning. BERT pre-training is used to train the BERT model itself which is then used for downstream tasks (that’s where it is fine-tuned). Very few people (researchers) are doing BERT pre-training and developers mostly use pre-trained models available on HF hub for their particular tasks. This is where the labels likely become relevant. In BERT pre-training there are no labels, it’s an unsupervised training task.