Thanks for the explanations.
I’m looking at the code to further understand how the loss is calculated under the hood.
In the Bert paper it mentions

the final hidden vectors corresponding to the mask tokens are fed into an output softmax over the vocabulary,

I’m looking at the code, and it seems to have components not mentioned in the paper.
I see that the hidden states first go through a transformation

  
      github.com
  
  
    huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L523 11

    def forward(self, hidden_states):        # We "pool" the model by simply taking the hidden state corresponding        # to the first token.        first_token_tensor = hidden_states[:, 0]        pooled_output = self.dense(first_token_tensor)        pooled_output = self.activation(pooled_output)        return pooled_outputclass BertPredictionHeadTransform(nn.Module):    def __init__(self, config):        super().__init__()        self.dense = nn.Linear(config.hidden_size, config.hidden_size)        if isinstance(config.hidden_act, str):            self.transform_act_fn = ACT2FN[config.hidden_act]        else:            self.transform_act_fn = config.hidden_act        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)    def forward(self, hidden_states):



  
  
    
    
  
  


And then I see that there is an entire separate matrix each of the tokens, here

  
      github.com
  
  
    huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L547 8

        return hidden_statesclass BertLMPredictionHead(nn.Module):    def __init__(self, config):        super().__init__()        self.transform = BertPredictionHeadTransform(config)        # The output weights are the same as the input embeddings, but there is        # an output-only bias for each token.        self.decoder = nn.Linear(config.hidden_size, config.vocab_size, bias=False)        self.bias = nn.Parameter(torch.zeros(config.vocab_size))        # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`        self.decoder.bias = self.bias    def forward(self, hidden_states):        hidden_states = self.transform(hidden_states)        hidden_states = self.decoder(hidden_states)        return hidden_states



  
  
    
    
  
  


which is multiplied by all the hidden states -> +bias.
Which makes sense since two seperate sets of embeddings (one for the inputs, one for the labels) is how word2vec was trained.
But I didn’t see these components in the paper. I’m guessing that they were copied the original Tensorflow version of Bert?
Edit:
I’m looking at the original Tensorflow version of Bert, and they use the same embedding matrix for the inputs and the labels
github dot com /google-research/bert/blob/master/run_pretraining.py#L141
github dot com / google-research/bert/blob/master/modeling.py#L409
nvidia also uses one embedding matrix for both
github dot com /NVIDIA/DeepLearningExamples/blob/ad49eae34e97b533a7fefb1c5ec10bbfa5cc4256/PyTorch/LanguageModeling/BERT/modeling.py#L551