The implementation of GPT2 masked attention mechanism will cause errors when the model was trained after some iterations.
# 🐛 Bug

## Information
Model I am using (GPT2):
Language I am using the model on ( Chinese ...): 

File in modeling_gpt2.py：
146        w = w * b - 1e4 * (1 - b)  # here the bias "-1e4" is too big, suggest replace it to "-1e10"

The item value of "query * key" may be close to or lower to "-1e4" ,  then after the softmax operation, the weights of masked attention may be attend to "unseen" context. This will cause inference errors when to evaluate the model.

Thanks for the bug report @xunzi2020! Did you by any chance measure the degree results would differ when replacing `-1e4` by `-1e10`? <|||||>Yes, I trained with chinese text (~30GB), it happened.


Patrick von Platen <notifications@github.com> 于2020年3月9日周一 下午4:11写道：

> Thanks for the bug report @xunzi2020 <https://github.com/xunzi2020>! Did
> you by any chance measure the degree results would differ when replacing
> -1e4 by -1e10?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/3181?email_source=notifications&email_token=AOYV777JVZ6HL4BNOKYY7HDRGSQEXA5CNFSM4LEAB3C2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOGCLSI#issuecomment-596387273>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AOYV777AZV3K5GBW4TFVFFDRGSQEXANCNFSM4LEAB3CQ>
> .
>
<|||||>Do you have a comparison such as: 

| - | GPT2 with -1e4 vs.  GPT2 with -1e10 | 
| ------------- | ------------- | 
| abs mean(softmax(logits) - softmax(logits)) | ???? |
| relative mean(softmax(logits) - softmax(logits)/softmax(logits))  | ???? |

lets say averaged over 100 - 1000 input samples and all GPT2 logits (50256)? 

That would be great to quantify the impact this change would have.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
