GPT2: Perfect training and evaluation loss, but terrible test-time performance
### System Info

I get the following error when running `transformers-cli env`:

```shell                                                                                                                                                                                                                                                                                       
Traceback (most recent call last):
  File "/Users/micah/opt/anaconda3/envs/fBERT/bin/transformers-cli", line 7, in <module>
    from transformers.commands.transformers_cli import main
  File "/Users/micah/opt/anaconda3/envs/fBERT/lib/python3.7/site-packages/transformers/commands/transformers_cli.py", line 26, in <module>
    from .user import UserCommands
  File "/Users/micah/opt/anaconda3/envs/fBERT/lib/python3.7/site-packages/transformers/commands/user.py", line 20, in <module>
    from huggingface_hub.hf_api import HfFolder, create_repo, list_repos_objs, login, logout, whoami
ImportError: cannot import name 'list_repos_objs' from 'huggingface_hub.hf_api' (/Users/micah/opt/anaconda3/envs/fBERT/lib/python3.7/site-packages/huggingface_hub/hf_api.py)
```

`4.16.2` is the Transformers library version I'm using. I don't think any other info is relevant.



### Who can help?

Copying from the `GPT2` list above: @patil-suraj, @patrickvonplaten, @LysandreJik 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I encountered a pretty ridiculous failure mode (with my own training pipeline, and no modification to the transformers library) in which, using GPT2 (imported as `from transformers import GPT2Config, GPT2Model`):
- I was getting almost 0 training and validation loss
- I was getting very bad performance when feeding the model incomplete sequences (e.g. for test-time word-generation)

After much debugging, I found that the issue was that the value of `self.masked_bias` (currently set to `-10e4`) is too low, which is supposed to implement the "mask" of the causal masked attention. 

For some high enough learning rates (which are still stable, however), the network is able to find a hack to copy the input to the output (getting around the causal masking): just drive the attention weights lower than `-10e-4`, and the causal mask will effectively not be doing anything! This means that the model will be able to attend the whole input in trying to generate the output (even the future tokens), so it will be able to simply copy it at training and validation time (getting 0 loss).

What I found while debugging [this line](https://github.com/huggingface/transformers/blob/215e0681e4c3f6ade6e219d022a5e640b42fcb76/src/transformers/models/gpt2/modeling_gpt2.py#L207) (before the softmax call):
```
>>> attn_weights[0,0,:4,:4]
[[-15258.9805, -10000.0000, -10000.0000, -10000.0000],
        [-15044.7910, -16940.4766, -10000.0000, -10000.0000],
        [-11722.1553, -13301.4287,  -1438.0649, -10000.0000],
        [ -9711.6445, -11315.6006,  -1065.3066, -12052.6035]]
```

As one can see, the attention weights outside of the causal masking are even larger than this fixed value, which lead the softmax to return non-zero weights on all inputs. The off-upper-triangle entries should be much larger, so that the causal-mask-triangle entries get assigned 0 weight.

### Expected behavior

In terms of fixes, there should probably be an assertion error that the weights never go below `self.masked_bias`, or setting it to an even lower value. Alternatively, it could return NaNs? It should definitely not fail silently.

The original code by Andrej Kaparthy didn't have this issue, as it effectively set `self.masked_bias` to `float('-inf')`, as can be seen [here](https://github.com/karpathy/minGPT/blob/3ed14b2cec0dfdad3f4b2831f2b4a86d11aef150/mingpt/model.py#L71).<|||||>Hey @micahcarroll,

Thanks for reporting this! Your issue might have tipped over the domino that solves this issue :sweat_smile: 

See: https://github.com/huggingface/transformers/issues/14859<|||||>@micahcarroll 

Thanks for reporting, I am working on it. I am wondering if you trained from scratch or fine-tune the original GPT2 model?
Also could you share what LR you used (as you mentioned `For some high enough learning rates`).

Thanks in advance!
<|||||>Thanks @ydshieh!

I was training from scratch on a sequential decision problem task (as in [Decision Transformers](https://github.com/kzl/decision-transformer)), with a learning rate of 1e-3.<|||||>Hi @micahcarroll 

So far I wasn't able to reproduce the issue. I tried to train a causal LM model from scratch using GPT2 (without the pretrained weights). I did get some large weights with magnitude in -1000 ~ -4000 though.

Despite this is not absolutely necessary, and we believe the fix will work (i.e. use a very large negative value), it would be super if I can reproduce the issue, and verify that our approach will indeed avoid it without any new issue.

Is there any chance you can share your training script, training arguments, and possibly the datasets you use (if it is not confidential).<|||||>The code hasn't yet been licensed for public release (nor it will be at least until September).

However, I completely agree with you that using the largest negative value possible should be sufficient to fix the error. Especially if you can use "inf" in some form (`np.inf` or `float('inf')` as was done in the original minGPT code, linked in prior comment).<|||||>@micahcarroll Understand, thank you!

Fortunately, I was able to get this issue by training a T5 model, so I can see if the fix would prevent such situation :) <|||||>@micahcarroll This is fixed. It would be really appreciated if you can also verify on your side :-)
I am going to close the issue.