Fine Tuned GPT2 model performs very poorly on token classification task
## Environment info

- `transformers` version: 4.17.0.dev0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.10.0+cu111 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

###Models:
- GPT-2: @patrickvonplaten, @LysandreJik

## Information

Model I am using gpt2:

The problem arises when using:
* [run_ner ] the official example scripts:

The tasks I am working on is:
Token classification on ncbi_disease dataset

## To reproduce
python run_ner.py \
  --model_name_or_path gpt2 \
  --dataset_name ncbi_disease \
  --output_dir /tmp/test-ner \
  --do_train \
  --do_eval
The f1-score after fine-tuning for 3 epoch is only 0.6114 whereas roberta-base provides f1 score of 0.8358 on the evaluation dataset. 
Hey @Dhanachandra,

As this is not an obvious bug, could you please try to make use of the forum: https://discuss.huggingface.co/ here instead?<|||||>Sorry, @patrickvonplaten, 
Due to curiosity to know the reason, I was posting the querry here. <|||||>That's totally fine! I hope you'll find some help on the forum :-)<|||||>Just commenting on this since I was trying to do a similar thing. I unfortunately had some false-confidence since the code specifically checks for `gpt2` config. Hopefully this will help people looking at this in the future! 

There is a subtle oversight in the code for decoder-only models. The `run_ner.py` code puts the label on the first of tokenized inputs. So for example:
```
Inputs: "He is on the ketogenic diet"
Label: "ketogenic diet" -> treatment
Tokenized-inputs: ["He", "is", "on", "the", "ke", "to", "genic", "diet"]
Old Model Labels: [0, 0, 0, 0, 1, -100,  -100,  1]
```

So it will only see "ke" before needing to make a prediction. This should be changed to the following:
```
Tokenized-inputs: ["He", "is", "on", "the", "ke", "to", "genic", "diet"]
New Model Labels: [0, 0, 0, 0, -100, -100, 1, 1]
```