I am new to the topic of Transformers, but my understanding (limited as it is) of BERT is that whilst the pre-trained embedding is created through semi-supervised learning (just large unannoted text corpora), when it comes to fine-tuning BERT for a specific task then this is usually a supervised learning process. So effectively you tell it what the right answers are and as part of the learning process in fine-tuning it will learn which contextualised embedding (and maybe a lot of other related knowledge) is relevant to what you want it to do.