Value error while running run_glue.py example with gpt2 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.7.0
- Platform: Linux
- Python version: 3.8.1
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.3 , GPU : yes
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:no

Tagging people:
@patrickvonplaten, @LysandreJik, @sgugger, @patil-suraj

## Information
Model I am using (Bert, XLNet ...): GPT2

The problem arises when using:
* [ ] the official example scripts: tensorflow/run_glue.py

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) GLUE (MNLI/SST2)

## Error
 ValueError: Dimension size must be evenly divisible by 192 but is 8 for '{{node sparse_categorical_crossentropy_2/Reshape_2}} = Reshape[T=DT_FLOAT, Tshape=DT_INT32](sparse_categorical_crossentropy_2/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits, sparse_categorical_crossentropy_2/strided_slice_1)' with input shapes: [8], [4] and with input tensors computed as partial shapes: input[1] = [2,8,12,?].

## To reproduce
python run_glue.py --model_name_or_path gpt2 --task_name mnli --do_train --do_eval --do_predict --output_dir ./output

## Expected behavior
Successfully complete training

Hello! Is this the exact command you're using? I tried to reproduce but I'm getting an error with the pad token which is not defined in the GPT-2 tokenizer. Did you tweak your GPT-2 tokenizer in order to add a padding token?<|||||>That's the exact command I am running. The only change I did (see below) was  to comment out clipnorm, to fix the error "ValueError: Gradient clipping in the optimizer (by setting clipnorm or clipvalue) is currently unsupported when using a distribution strategy."  

-            clipnorm=training_args.max_grad_norm,
+            #clipnorm=training_args.max_grad_norm,<|||||>@LysandreJik  any luck reproducing the error?<|||||>When I set the dataset_mode to constant_batch, I see the following error. Any idea why the logits output dimension is (batch_size, sequence_length, num_labels) and not (batch_size, num_labels) ?

    ValueError: Shape mismatch: The shape of labels (received (8, 1)) should equal the shape of logits except for the last dimension (received (8, 128, 3)).<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.