Masked LM implementation details
I read the source code of `TFBertMLMHead`, it seems that, this layer just predict the whole sequence, rather than predict the `MASKED` tokens.

`TFBertMLMHead` just do these things:

* transform the `hidden state` from the last encoder layer
* `predictions = tf.matmul(hidden_state, input_embedding_matrix)`, with shape (batch_size, sequence_length, vocab_size)
* return the `predictions` to calculate loss

The inputs are simply the `hidden state` of the last encoder layer.

But the implemenmtation from [google-research/bert](https://github.com/google-research/bert/blob/cc7051dc592802f501e8a6f71f8fb3cf9de95dc9/run_pretraining.py#L240) needs extra inputs `masked_lm_positions` and `masked_lm_weights`, and then use these inputs to calculate the masked lm loss.

So, does the `TFBertMLMHead` miss something?

The method from google-research/bert you're showing returns the loss. We don't return the loss with `TFBertForMaskedLM`, we return the output distribution over the vocabulary. You can then use this output distribution to compute the loss, with a cross entropy loss for example.<|||||>I see, thanks.