Issues with TFGPT2ForSequenceClassification
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.5.1
- Platform: Google Colab
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.4.1
- Using GPU in script?: NO, but tf automatically use it  
- Using distributed or parallel set-up in script?:

### Who can help
@patrickvonplaten, @LysandreJik, @Rocketknight1


## Information

Model I am using (GPT2):

The problem arises when using:
* [ ] my own modified scripts: (give details below)
When using TFGPT2ForSequenceClassification, I found that the structure of the model is weird, see below:
![image](https://user-images.githubusercontent.com/38811872/116583073-95d4fd80-a8db-11eb-8868-be42e11cdf08.png)
Why is the classifier inserted before the GPT main layer? And when I load the PyTorch version, it looks different (inserted after the main layer): 
![image](https://user-images.githubusercontent.com/38811872/116583497-08de7400-a8dc-11eb-945b-2477be9aa852.png)
Also, I tried to train this model as the tutorials of [fine-tuning on Bert with customized dataset](https://huggingface.co/transformers/custom_datasets.html) suggests, but failed as following, I loaded the pretrained classification model with 3 classes: 
ValueError: in user code:

    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:805 train_function  *
        return step_function(self, iterator)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:795 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/distribute/distribute_lib.py:1259 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/distribute/distribute_lib.py:2730 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/distribute/distribute_lib.py:3417 _call_for_each_replica
        return fn(*args, **kwargs)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:788 run_step  **
        outputs = model.train_step(data)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:758 train_step
        self.compiled_metrics.update_state(y, y_pred, sample_weight)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/compile_utils.py:408 update_state
        metric_obj.update_state(y_t, y_p, sample_weight=mask)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/metrics_utils.py:90 decorated
        update_op = update_state_fn(*args, **kwargs)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/metrics.py:177 update_state_fn
        return ag_update_state(*args, **kwargs)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/metrics.py:618 update_state  **
        matches = ag_fn(y_true, y_pred, **self._fn_kwargs)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/metrics.py:3315 sparse_categorical_accuracy
        return math_ops.cast(math_ops.equal(y_true, y_pred), K.floatx())
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/math_ops.py:1679 equal
        return gen_math_ops.equal(x, y, name=name)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/gen_math_ops.py:3179 equal
        name=name)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/op_def_library.py:750 _apply_op_helper
        attrs=attr_protos, op_def=op_def)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/func_graph.py:592 _create_op_internal
        compute_device)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py:3536 _create_op_internal
        op_def=op_def)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py:2016 __init__
        control_input_ops, op_def)
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py:1856 _create_c_op
        raise ValueError(str(e))

    ValueError: Dimensions must be equal, but are 3 and 512 for '{{node Equal}} = Equal[T=DT_FLOAT, incompatible_shape_error=true](Cast_1, Cast_2)' with input shapes: [?,3], [?,512].
   
The tasks I am working on is:
* [ ] my own task or dataset: (give details below)
The task is a multi-label classification task, where the label of each sample could be represented as a 3-dim vector like [0,0,0], [0,1,0], [1,1,0], etc. 

## To reproduce

Steps to reproduce the behavior:

1. load the GPT2Tokenizer, TFGPT2ForSequenceClassification with num_labels=3
```
my_gpt_tokenizer = GPT2TokenizerFast.from_pretrained('openai-gpt')
my_gpt_model = TFGPT2ForSequenceClassification.from_pretrained('openai-gpt',num_labels=3)
```
2. add pad token to the tokenizer, tokenize the text as the tutorials did and transfer them into dataset objects  
```
my_gpt_tokenizer.add_special_tokens({'pad_token': '[PAD]'})
gpt_train_encodings = my_gpt_tokenizer(X_train, truncation=True, padding=True)
gpt_test_encodings = my_gpt_tokenizer(X_test, truncation=True, padding=True)
gpt_train_dataset = tf.data.Dataset.from_tensor_slices((dict(gpt_train_encodings),y_train))
gpt_test_dataset = tf.data.Dataset.from_tensor_slices((dict(gpt_test_encodings),y_test))
```
3. train the model:
```
optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)
my_gpt_model.compile(optimizer=optimizer, loss="binary_crossentropy", metrics=['accuracy']) 
history = my_gpt_model.fit(gpt_train_dataset.shuffle(500).batch(10), epochs=2, batch_size=10, validation_data=gpt_test_dataset.batch(10))
```
 

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
The model should be trained successfully as the Bert classification does.  I tried the same code on TFBertForSequenceClassification and TFDistilBertForSequenceClassification, which are all successful. 
<!-- A clear and concise description of what you would expect to happen. -->

Thanks for the very in-detail issue description! @Rocketknight1 do you maybe want to give it a try here? Otherwise I'm happy to take a look :-)<|||||>Taking a look now!<|||||>Hi @cytwill, can you share a few lines of the data you're loading as X_train and y_train? If it's a private dataset, you can replace the text with random text - I just want to see the format of the data and try to reproduce the error here.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi. I am currently experiencing the same issue as the OP where the classification layer seems to be inserted before the main GPT layer. I basically have the same model summary and a similar error so I thought I'd try to reopen this.

I know it's not an ideal dataset for the model but here's a copy of the Fine Tuning with Keras tutorial to illustrate the problem: https://colab.research.google.com/drive/1UJdB5QG_6L1qeWxM8Fa-CuDZQR32cshL?usp=sharing

Below the tensorflow implementation is the pytorch version that seems to work well enough.