Cuda out of memory when fp16 training
# ❓ Questions & Help
I have error when run train with fp16
<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
I want to train Albert model on Russian. I use run_language_modeling.py with some changes in dataset creating. If i run training with fp32 (without --fp flag) and batch_size=2, I will ok. But if i run training with --fp16 flag, i will obtain cuda out of memory error. I tried to decdeare batch size to 1, however, I obtained same error.

Same issue when running distilbart<|||||>I face the same issue with PEGASUS<|||||>I've run into this with several seq2seq models (Pegasus, BART, T5). Ironically, running with fp16 causes increased GPU memory usage.<|||||>I solved my issue by downgrading Pytorch to 1.5.1 and installing Nvidia/Apex. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
