Model conversion from PyTorch to TF2 doesn't work properly for ALBERT
# 🐛 Bug

## Information

The model conversion script `convert_pytorch_checkpoint_to_tf2.py` seems to be not working properly for ALBERT models.
It fails on the pre-trained models officially released by Google which are converted to PyTorch models with `convert_albert_original_tf_checkpoint_to_pytorch.py`.

## To reproduce

```
$ wget https://storage.googleapis.com/albert_models/albert_base_v2.tar.gz
$ tar xzf albert_base_v2.tar.gz
$ cd albert_base/
$ python -m transformers.convert_albert_original_tf_checkpoint_to_pytorch --tf_checkpoint_path model.ckpt-best --pytorch_dump_path ./pytorch_model.bin --albert_config_file albert_config.json
$ python -m transformers.convert_pytorch_checkpoint_to_tf2 --tf_dump_path ./ --model_type albert --pytorch_checkpoint_path ./pytorch_model.bin --config_file albert_config.json --compare_with_pt_model
...
Max absolute difference between models outputs 17.709423065185547
Traceback (most recent call last):
  File "/home/m-suzuki/.pyenv/versions/3.7.4/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/m-suzuki/.pyenv/versions/3.7.4/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/m-suzuki/.pyenv/versions/Python-3.7.4/lib/python3.7/site-packages/transformers/convert_pytorch_checkpoint_to_tf2.py", line 499, in <module>
    only_convert_finetuned_models=args.only_convert_finetuned_models,
  File "/home/m-suzuki/.pyenv/versions/Python-3.7.4/lib/python3.7/site-packages/transformers/convert_pytorch_checkpoint_to_tf2.py", line 428, in convert_all_pt_checkpoints_to_tf
    compare_with_pt_model=compare_with_pt_model,
  File "/home/m-suzuki/.pyenv/versions/Python-3.7.4/lib/python3.7/site-packages/transformers/convert_pytorch_checkpoint_to_tf2.py", line 351, in convert_pt_checkpoint_to_tf
    assert diff <= 2e-2, "Error, model absolute difference is >2e-2: {}".format(diff)
AssertionError: Error, model absolute difference is >2e-2: 17.709423065185547
```

Same error for ALBERT v1 models.

## Expected behavior

Max absolute difference between models outputs should be <= 2e-2

## Environment info

Observed on both of the following environments: 

- `transformers` version: 2.5.1
- Platform: Darwin-19.3.0-x86_64-i386-64bit
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
<!-- -->
- `transformers` version: 2.5.1
- Platform: Linux-4.15.0-58-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Proposed a fix that didn't work out-of-the-box with official ALBERT models. Still looking into it, will keep you posted.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Closing since it seems solved in #4076. Thank you!