Fix gradient overflow issue during attention mask
This fix is in reference to issue #382. GPT2 can now be trained in mixed precision, which I've confirmed with testing. I also tested unconditional generation on multiple seeds before and after changing 1e10 to 1e4 and there was no difference. Please let me know if there is anything else I can do to make this pull request better. Thanks for all your work!
Ok, great, thanks @SudoSharma!<|||||>While the outputs are the same between 1e10 and 1e4, I shouldn't expect the outputs between fp32 and fp16 to be the same, should I? I get different outputs between the two when doing unconditional/conditional generation with top_k=40 but even with top_k=1. Usually they're the same for a while and then deviate. This is with Apex installed, so using FusedLayerNorm.

If I turn on Apex's AMP with `from apex import amp; amp.init()` then they still deviate but after a longer time (I think it makes the attention nn.Softmax use fp32). Have to remove the `model.half()` call when using AMP.

Perhaps it's not realistic to have the outputs be the same when fp16 errors in the "past" tensors are compounding as the sequence gets longer? But it is surprising to see them differ for top_k=1 (deterministic) since only the largest logit affects the output there.

P.S. For my site it's been enormously helpful to have this PyTorch implementation. @thomwolf Thank you!<|||||>Hi @AdamDanielKing,
Congratulation on your demo!
Are you using the updated API for apex Amp? (https://nvidia.github.io/apex/amp.html)
Also, we should discuss this in a new issue? At first, I thought this was related to this PR but I understand it's not, right?<|||||>@thomwolf You're probably right that a new issue is best. I've created one at #602.

Thanks for pointing out I was using the old Apex API. Switching to the new one unfortunately didn't fix the issue though.