GPT2 MNLI training using run_glue.py
Running this on Google Colab,

```
!python run_glue.py \
  --model_name_or_path gpt2 \
  --task_name mnli \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 10 \
  --gradient_accumulation_steps 32\
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir models/gpt2/mnli/
```

I get the following error,

```
 "Asking to pad but the tokenizer does not have a padding token. "
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
``` 
@LysandreJik : does the trainer need to be modified? Or am I supposed to be using additional commands for training using GPT2? (I have been following [this example](https://huggingface.co/transformers/v2.0.0/examples.html) using BERT)

As explained in the documentation: "`run_glue.py`: This script can fine-tune the following models: BERT, XLM, XLNet and RoBERTa."

=> GPT-2 is a Transformer decoder, which can learn to generate text in an autoregressive way. It is not aimed at GLUE tasks, which are sequence classification tasks. <|||||>Hi! Actually we've recently added `GPT2ForSequenceClassification` to enable support for sequence classification tasks (like GLUE). The support was added to enable some models such as EDIT: linked wrong model. Updated: [DialogRPT](https://huggingface.co/microsoft/DialogRPT-updown)!

However, as you have seen @nlp-student, the GPT-2 model isn't trainable out of the box with batch size > 1, as it has no padding token defined. Furthermore, MNLI is a three-way classification so you would need to set the number of labels appropriately.

I invite you to run the following script to create a model that you can then use with the `run_glue.py` script, initialized from the GPT-2 weights:
```py
from transformers import GPT2ForSequenceClassification, GPT2Tokenizer

model = GPT2ForSequenceClassification.from_pretrained("gpt2", num_labels=3)
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

# Define a padding token
tokenizer.pad_token = tokenizer.eos_token
model.config.pad_token_id = tokenizer.pad_token_id

# Save the model and tokenizer in a directory
model.save_pretrained("directory")
tokenizer.save_pretrained("directory")
```

Then you can launch the `run_glue.py` script by specifying that checkpoint.

However, as @NielsRogge pointed out, GPT-2 will not obtain as good results as a bi-directional encoders such as BERT.<|||||>Thank you so much for your responses. I will try that out.

Is GPT-2 generally expected to perform worse than BERT on sequence classification ? 

I've seen a lot of examples of BERT finetuned for classification, but not really for GPT-2. Is it not common (or that useful) to finetune GPT-2 for classification?

Thank you for any help or direction!<|||||>> Is GPT-2 generally expected to perform worse than BERT on sequence classification ?

Normally, yes. The reason for this is that GPT-2 is not really designed for such a task, whereas BERT is, as it has a special [CLS] token for classification tasks. GPT-2 is designed to process text autoregressively (i.e. left to right) in order to generate new text, whereas BERT is designed to process all tokens at once (hence creating a bidirectional representation of all input tokens), which is useful for tasks like sequence classification or extractive question answering for example. 

That said, you can still use GPT-2 to perform sequence classification. You can simply let GPT-2 process a sentence word by word from left to right, and then train it to predict the class of the sentence by placing a linear layer on top of the hidden representation of the final token of the sentence, which is done by looking at the [code](https://github.com/huggingface/transformers/blob/285c6262a84490270d2f1a1c06ee9ccfc1b60e8f/src/transformers/models/gpt2/modeling_gpt2.py#L1233) of `GPT2ForSequenceClassification`.

 So maybe an interesting thing to do is compare `BERTForSequenceClassification` and `GPT2ForSequenceClassification` on the same dataset, and see which one performs best. <|||||>Many thanks for your detailed response! It was very helpful for my understanding.<|||||>You're welcome, I'm closing this issue! Feel free to reopen if you have other issues down the road.<|||||>> Hi! Actually we've recently added `GPT2ForSequenceClassification` to enable support for sequence classification tasks (like GLUE). The support was added to enable some models such as EDIT: linked wrong model. Updated: [DialogRPT](https://huggingface.co/microsoft/DialogRPT-updown)!
> 
> However, as you have seen @nlp-student, the GPT-2 model isn't trainable out of the box with batch size > 1, as it has no padding token defined. Furthermore, MNLI is a three-way classification so you would need to set the number of labels appropriately.
> 
> I invite you to run the following script to create a model that you can then use with the `run_glue.py` script, initialized from the GPT-2 weights:
> 
> ```python
> from transformers import GPT2ForSequenceClassification, GPT2Tokenizer
> 
> model = GPT2ForSequenceClassification.from_pretrained("gpt2", num_labels=3)
> tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
> 
> # Define a padding token
> tokenizer.pad_token = tokenizer.eos_token
> model.config.pad_token_id = tokenizer.pad_token_id
> 
> # Save the model and tokenizer in a directory
> model.save_pretrained("directory")
> tokenizer.save_pretrained("directory")
> ```
> 
> Then you can launch the `run_glue.py` script by specifying that checkpoint.
> 
> However, as @NielsRogge pointed out, GPT-2 will not obtain as good results as a bi-directional encoders such as BERT.

Hi,  met the same error as the original post although but I've set the per_device_train_batch_size=1. 
I was running run_swag.py with GPT2 (I added a class GPT2ForMultipleChoice referring to BertForMultipleChoice), could you offer some help with this problem?　Thanks a lot.<|||||>I follow the suggestion but got this error:

RuntimeError: Error(s) in loading state\_dict for GPT2ForSequenceClassification: size mismatch for score.weight: copying a param with shape torch.Size(\[100, 768\]) from checkpoint, the shape in current model is torch.Size(\[2, 768\]). You may consider adding \`ignore\_mismatched\_sizes=True\` in the model \`from\_pretrained\` method.

I then tried to flag `ignore_mismatched_sizes` as True like below but still, get the same error. 

```python
from transformers import GPT2ForSequenceClassification, GPT2Tokenizer

model = GPT2ForSequenceClassification.from_pretrained("gpt2", num_labels=100, ignore_mismatched_sizes=True)
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

# Define a padding token
tokenizer.pad_token = tokenizer.eos_token
model.config.pad_token_id = tokenizer.pad_token_id

# Save the model and tokenizer in a directory
model.save_pretrained("directory")
tokenizer.save_pretrained("directory")
```