fp16 overflow in GPT-2
When trying to train in mixed precision, after casting model weights to fp16 overflow is bound to occur since multiplication by 1e10 is used to mask the attention weights.

I noticed BERT multiplies by 1e4 (within fp16 range) instead, and the overflow problem doesn't occur and now it's training happily :)

I'm happy to make the various changes and PR if wanted?
Hi @andrewPoulton, yes indeed we could update that for GPT-2, would be happy to get a PR.
Can you check the generations are identical for a few seeds (it should be)?<|||||>Yeah, sure - what generations do you mean?<|||||>Fixed with #495