No Causal Attention Masking in GPT-2 LM Finetuning Script
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT-2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
running run_language_modeling.py on WikiText-2 dataset

* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. attention_mask is None at each forward step of the GPT-2 model (GPT2LMHeadModel)


## Expected behavior
attention_mask should reflect causal attention masking for the LM objective in finetuning GPT-2 so that outputs (t) only attend to inputs at previous time steps (1,..,t-1) instead of relying on input at the same time-step of output (t) where GPT-2 can simply learn to copy the input as output to optimize the LM objective.



## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.5.1
- Platform: Linux-4.15.0-76-generic-x86_64-with-glibc2.10
- Python version: 3.8.1
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Hi @alvinchangw, 

GPT2 always uses causal masking no matter what kind of attention_mask you give it. 
It's easy to see when you print out the computed attentions for each layer (by setting `output_attentions=True`) => see for this also #2975. 

In the code this is done in this line:
https://github.com/huggingface/transformers/blob/298bed16a841fae3608d334441ccae4d9043611f/src/transformers/modeling_gpt2.py#L146

I admit it is very cryptic and probably should have better naming. Essentially what happens here is the following:
`self.bias` is defined as a lower triangular mask (see torch function [here](https://pytorch.org/docs/stable/torch.html?highlight=tril#torch.tril) ). according to the sequence length (params `nd` and `ns`), we derive `b`. `b` is then a lower triangular mask of shape sequence length x sequence length. Using this mask, we substract 10^4 from all values in `w` which should be masked, which sets their attention to 0.
<|||||>Hi @patrickvonplaten,

Thank you for pointing this out and for the detailed explanation! 