TF LMHead very slow: TFGPT2LMHeadModel is 7 times slower than Torch GPT2LMHeadModel 
The benchmark shows the Tensorflow TFGPT2LMHeadModel is 7 times (140ms) slower than Torch GPT2LMHeadModel implementation (20ms). Based on the profile tool from latest https://www.tensorflow.org/tfx/serving/tensorboard, the bottleneck is here. https://github.com/huggingface/transformers/blob/d9149f00d1a4650bafa7e1cd73e10398193c852c/src/transformers/modeling_tf_utils.py#L796

```
from transformers import *

args = PyTorchBenchmarkArguments(
    models=["gpt2_pt"],
    batch_sizes=[1], sequence_lengths=[8])

config_base = GPT2Config(n_layer=2, n_head=2)
config_base.architectures = ["GPT2LMHeadModel"]
benchmark = PyTorchBenchmark(args, configs=[config_base])
print(benchmark.run())

args = TensorFlowBenchmarkArguments(
    models=["gpt2_tf"],
    batch_sizes=[1], sequence_lengths=[8])

config_base = GPT2Config(n_layer=2, n_head=2)
config_base.architectures = ["GPT2LMHeadModel"]
benchmark = TensorFlowBenchmark(args, configs=[config_base])
print(benchmark.run())
```

Without LMHeadModel, the benchmark is 6ms vs 8ms.
Multiple issues may be related to this.
[My finetuned gpt2 model is taking wayy too long to generate samples, like 5-8 minutes] 
https://github.com/huggingface/transformers/issues/6173
[[Benchmark] TFGPT2LMHeadModel is five times slower than GPT2LMHeadModel]
https://github.com/huggingface/transformers/issues/5604
[tensorflow2_gpt2 Slow speed]
https://github.com/huggingface/transformers/issues/4634
[In Tensorflow the serving is very slow]
https://github.com/huggingface/transformers/issues/5341

Do we plan to solve this problem with a higher priority? Thanks. @patrickvonplaten 

Can we add a new linear layer instead of using wte with matmul? Like `self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)` in pytorch?
Since tf.matmul in CPU seems quite slow and moreover we have a `transpose_b` in the current TF implementation. <|||||>Thanks a lot for posting the issue and linking everything together - this is extremely useful! I will try to take a deeper look into it early next week! <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I met the same situation with the env info:
transformer version:4.12.5
tensorflow-gpu:2.8.0
tf-serving:2.6.0
linux, gpu-machine:P100, test script with the batch_size 1, max_len:20

google's bert inference takes about 12-13ms per query, but TF-Bert takes about 45~58ms with or without xla_cpu_compilation_enabled, 3-5x slower.

Has the problem been solved? Or some ideas to improve the TF-Pretrained-Model performemce?