Fix non-negligible difference between GPT2 and TFGP2
# What does this PR do?

Fix #13666 
The following test should have caught that error: `test_pt_tf_model_equivalence` in `modeling_tf_common.py` - would you like to check why it was passing even though it shouldn't have been?<|||||>> 
> 
> The following test should have caught that error: `test_pt_tf_model_equivalence` in `modeling_tf_common.py` - would you like to check why it was passing even though it shouldn't have been?

Sure. Will do it tomorrow :)<|||||>Thank you @ydshieh, this is very helpful.<|||||>Hi, @LysandreJik ,

In `test_modeling_tf_gpt2.py` , `TFGPT2ModelTester` uses a config with `"gelu"` activation.

https://github.com/huggingface/transformers/blob/8e908c8c74f556a82534f4cf1e7a1b4f7b55d24c/tests/test_modeling_tf_gpt2.py#L56

In `test_pt_tf_model_equivalence.py`, both tf & pytorch models are created using this same config, and therefore both use `"gelu"`.

Since the issue in `TFGPT2Model` was that it used `self.act = get_tf_activation("gelu")`, the test would pass.

The issue would be observed when the config uses an activation functions other than `"gelu"`, which is the case when I used the pretrained GPT2 (`"gelu_new"` is used).

https://huggingface.co/gpt2/blob/main/config.json

So `test_pt_tf_model_equivalence` is fine in my opinion, but it might be good to add other tests to prevent this situation (using pretrained models' configs, but reduce the hidden size & no. of layers).
(well, this doesn't 100% guarantee, if an attribute in a pretrained config happens to be the same as a hard coded attribute in a model).<|||||>@ydshieh, thank you for your detailed analysis, this is perfect! I understand the limitation of the test and this edge case.

Merging this PR!