GPT2: masked_bias should be sufficiently small instead of -1e4
See also #9594

My package build `transformers==4.12.3`.

Ideally for a causal language model, we should set it as -inf. Current setting -1e4 is not **small** enough. In my case, the model would learn to lower the attention weight of left context such that it could get the information of right context, which is illegal behavior.

Codes at https://github.com/huggingface/transformers/blob/531336bbfd2a97cf800f610d971d6ec0a1578752/src/transformers/models/gpt2/modeling_gpt2.py#L205-L206

In my experiment, I print the `attn_weights` of the first layer, there're many weights smaller than -1e4, and the attention score is no longer lower triangle, a.k.a., not causal.

```
tensor([[[[-16012.8271, -10000.0000, -10000.0000, -10000.0000, -10000.0000,
           -10000.0000, -10000.0000],
          [ 10511.5508,   5376.9136, -10000.0000, -10000.0000, -10000.0000,
           -10000.0000, -10000.0000],
          [-24461.1621, -13598.0195, -20461.9980, -10000.0000, -10000.0000,
           -10000.0000, -10000.0000],
          [-16113.5645, -10553.2480, -13133.3857,    147.0204, -10000.0000,
           -10000.0000, -10000.0000],
          [-20674.4199,  -2806.1458, -15805.4199,  -4623.0571,  12110.2920,
           -10000.0000, -10000.0000],
          [-24806.1738, -14453.9990, -20414.9160,  -8439.6494,  -2858.4604,
           -17795.5215, -10000.0000],
          [ -5393.4922, -10265.1875,  -6821.3672,   3551.4062, -20622.5879,
            -3372.0925,  10511.3047]],
```
Hey @maxwellzh, this is a similar conversation to what was said here: https://github.com/huggingface/transformers/issues/14859

Thanks for raising this issue!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.