Revert "[GPT2] Propose fix for #21080"
Reverts huggingface/transformers#21853

A few PT/TF and PT/Flax cross tests started to fail after #21853. Revert that PR for now. We need to look what went wrong (and why it is not reported in the PR CI)

```
FAILED tests/models/gpt2/test_modeling_tf_gpt2.py::TFGPT2ModelTest::test_pt_tf_model_equivalence - AssertionError: 3.225274 not less than or equal to 1e-05 : outputs.last_hidden_state: Difference between torch and tf is 3.225274085998535 (>= 1e-05).
FAILED tests/models/encoder_decoder/test_modeling_tf_encoder_decoder.py::TFGPT2EncoderDecoderModelTest::test_pt_tf_model_equivalence - AssertionError: 0.3753911 not less than or equal to 1e-05 : outputs.logits: Difference between torch and tf is 0.3753910958766937 (>= 1e-05).
FAILED tests/models/vision_encoder_decoder/test_modeling_tf_vision_encoder_decoder.py::TFViT2GPT2EncoderDecoderModelTest::test_pt_tf_model_equivalence - AssertionError: 0.42845678 not less than or equal to 1e-05 : outputs.logits: Difference between torch and tf is 0.42845678329467773 (>= 1e-05).
```

A job run with failed tests is [here](https://app.circleci.com/pipelines/github/huggingface/transformers/59606/workflows/fd5cb028-f3df-4ac6-83ae-b0cc68396af8/jobs/728991)


Tested locally - all good now. Will merge once CI is green too (already discussed offline)