Pretrained TFAlbertForMaskedLM returns seemingly random token predictions
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BERT, ALBERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task:
* [x] my own task or dataset: toy data.

## To reproduce

```
import tensorflow as tf
from transformers import BertTokenizer, TFBertForMaskedLM, AlbertTokenizer, TFAlbertForMaskedLM

tf.random.set_seed(1)
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = TFBertForMaskedLM.from_pretrained("bert-base-uncased")
input_ids = tokenizer.encode(f"I {tokenizer.mask_token} you", return_tensors="tf")
outputs = model(input_ids)
prediction_scores = outputs[0]
predicted_ids = tf.reshape(tf.argmax(prediction_scores, -1), [-1])
predicted_tokens = tokenizer.convert_ids_to_tokens(predicted_ids)
print(predicted_tokens)
# ['.', 'i', 'love', 'you', '.']


tf.random.set_seed(1)
tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")
model = TFAlbertForMaskedLM.from_pretrained("albert-base-v2")
input_ids = tokenizer.encode(f"I {tokenizer.mask_token} you", return_tensors="tf")
outputs = model(input_ids)
prediction_scores = outputs[0]
predicted_ids = tf.reshape(tf.argmax(prediction_scores, -1), [-1])
predicted_tokens = tokenizer.convert_ids_to_tokens(predicted_ids)
print(predicted_tokens)
# ['_pawn', '_addressing', '_fundraising', '_george', '_hybrid']
```

## Expected behavior

I would expect both commands to return the same result, filling in the middle with "love" or some other word. BERT performs correctly, while ALBERT seems to return nonsense. Any idea why this is happening?

## Environment info
- `transformers` version: 2.4.1
- Platform: Linux
- Python version: 3.6.5
- PyTorch version (GPU?): not installed
- Tensorflow version (GPU?): 2.0.0 (True)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Hi, thank you for opening an issue, there was indeed an error with the way the `TFAlbertModel` was implemented! It was fixed with https://github.com/huggingface/transformers/commit/1abd53b1aa2f15953bbbbbfefda885d1d9c9d94b.

Even with the fix, the sequence `I <mask> you` is hard for ALBERT, but using your sample with a longer sequence yields satisfying results:

```py
import tensorflow as tf
from transformers import BertTokenizer, TFBertForMaskedLM, AlbertTokenizer, TFAlbertForMaskedLM

tf.random.set_seed(1)
tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")
model = TFAlbertForMaskedLM.from_pretrained("albert-base-v2")
input_ids = tokenizer.encode(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.", return_tensors="tf")
outputs = model(input_ids)
prediction_scores = outputs[0]
predicted_ids = tf.reshape(tf.argmax(prediction_scores, -1), [-1])
predicted_tokens = tokenizer.convert_ids_to_tokens(predicted_ids)
print(predicted_tokens)
# ['▁time', '▁this', '▁is', '▁the', '▁best', '▁thing', '▁i', "'", 've', '▁done', '▁in', '▁my', '▁life', '!!!', '▁your']
```

Let me know if the updated model works for you.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
