Question regarding crossentropy loss function for BERTMaskedLM 
How does BERT handle large number of classes to predict? The number of classes is essentially the vocabulary size which is  30522 for the BERT-base model. When BERT tries to predict a word using CrossEntropy loss, it needs to compute the softmax for a large number of classes. 

In shallow approach such as word2vec, negative sampling or hierarchical softmax is used. I wonder why it is not the case for BERT.
Thanks in advance.
30k is ok for a softmax, it's not that much and that because Bert is using a sub-word (open-)vocabulary.

Full word (and closed-vocabulary) models like word2vec have to handle several 100k words hence the specific speed-ups. They are also older and the computation power available at the time was more constrained.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
