Bert, NLP.
Language Model” (MLM) and “Next Sentence Prediction” (NSP)). What we want is embeddings that encode the word meaning well… BERT is motivated to do this, but it is also motivated to encode anything else that would help it determine what a missing word is (MLM), or whether the second sentence came