DistilGPT2 extremely strange model behaviour
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Linux-4.15.0-91-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0a0+1606899 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes, 2
- Using distributed or parallel set-up in script?: Yes

### Who can help

 albert, bert, GPT2, XLM: @LysandreJik
 examples/distillation: @VictorSanh

## Information

Model I am using (Bert, XLNet ...): DistilGPT2

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ X ] my own modified scripts: (give details below)

Had to slightly modify the official scripts to even get it to run - other than that, effectively no difference

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X ] my own task or dataset: (give details below)

Chat logs. In the format

```
[22-Jul-20 04:15 AM] User
Message


[22-Jul-20 04:15 AM] Another user
Message


[22-Jul-20 04:16 AM] Etc.
You get the point
```


## To reproduce

Steps to reproduce the behavior:

1. (Fix and) run example scripts for distillation with a custom GPT2 model.

When trying to generate, the model will generate the same text with no change, regardless of using ``pipeline`` or ``model.generate``. If attempting to use input texts, the network will output the texts immediately appended by the phrase. In the two times I've run this experiment, it was the space character twice and ``[Nov-Nov-20] User`` once (after I tried removing lines that were under a certain token value)

## Expected behavior

The model to work as a normal transformer.

<!-- A clear and concise description of what you would expect to happen. -->

Extra info:

command: ``python3 -m torch.distributed.launch \
    --nproc_per_node=$N_GPU_NODE \
    --nnodes=$N_NODES \
    --node_rank $NODE_RANK \
    train.py \
        --force \
        --fp16 \
        --n_epoch 3 \
        --checkpoint_interval 2000 \
        --batch_size 12 \
        --n_gpu $WORLD_SIZE \
        --student_type gpt2 \
        --student_config training_configs/distilgpt2.json \
        --teacher_type gpt2 \
        --teacher_name gpt2-large \
        --freeze_pos_embs \
        --dump_path /root/distil/ \
        --data_file data/binarized_text.gpt2-large.pickle \
        --token_counts data/token_counts.gpt2-large.pickle``

Not using a custom tokenizer, just the gpt2-large tokenizer<|||||>When using the original `distilgpt2` checkpoint, can you generate coherent text?<|||||>> When using the original `distilgpt2` checkpoint, can you generate coherent text?

Yep, works perfectly. Maybe worth it to note that I was using 2x 3090 GPUs (I know how amperes can be sometimes)

Since my dataset is made up of a lot of `\n`, my first thought was that it might have just copied that to attempt to minimize loss, so I set the `lm_seq_dataset.py` to remove any lines under 4 tokens. It did change the string, but it makes absolutely no sense to me that it would learn to copy a string like this.

IMO (and I haven't really had the time to look into this so take it with a whole chunk of salt) this is probably a training issue, since the phrase is different on different data and the generation works on the default model. If it's a training issue, my best guess is that it might be iterating over the same line? But this is flawed because the line contains `[Nov-Nov`, which is incorrect.

Super strange bug all in all.<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.