Benchmarking on TPU shows clearly wrong results
# 🐛 Bug

## Information

I'm trying to benchmark performance of TPUs and the results don't make sense: they are the same for all batch sizes.

It was mentioned [in the pull request that added the feature](https://github.com/huggingface/transformers/pull/4850#issuecomment-640751636) but the PR was merged anyway.

## To reproduce

```
from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments

args = PyTorchBenchmarkArguments(
    models=["bert-large-uncased"],
    batch_sizes=[i * 1024 for i in range(2, 17)],
    sequence_lengths=[16],
    training=True,
    no_memory=True
)

benchmark = PyTorchBenchmark(args)
results = benchmark.run()
```

Output:

```
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
      bert-large-uncased            2048             16            0.027     
      bert-large-uncased            3072             16            0.029     
      bert-large-uncased            4096             16            0.028     
      bert-large-uncased            5120             16            0.027     
      bert-large-uncased            6144             16            0.027     
      bert-large-uncased            7168             16            0.028     
      bert-large-uncased            8192             16            0.028     
      bert-large-uncased            9216             16            0.027     
      bert-large-uncased           10240             16            0.027     
      bert-large-uncased           11264             16            0.027     
      bert-large-uncased           12288             16            0.027     
      bert-large-uncased           13312             16            0.027     
      bert-large-uncased           14336             16            0.027     
      bert-large-uncased           15360             16            0.028     
      bert-large-uncased           16384             16            0.028     
--------------------------------------------------------------------------------
TPU was used for inference. Note that the time after compilation stabilized (after ~10 inferences model.forward(..) calls) was measured.

====================        TRAIN - SPEED - RESULTS         ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
      bert-large-uncased            2048             16            0.089     
      bert-large-uncased            3072             16            0.074     
      bert-large-uncased            4096             16            0.091     
      bert-large-uncased            5120             16            0.091     
      bert-large-uncased            6144             16            0.091     
      bert-large-uncased            7168             16            0.075     
      bert-large-uncased            8192             16            0.089     
      bert-large-uncased            9216             16             0.09     
      bert-large-uncased           10240             16            0.074     
      bert-large-uncased           11264             16             0.09     
      bert-large-uncased           12288             16             0.09     
      bert-large-uncased           13312             16             0.09     
      bert-large-uncased           14336             16            0.077     
      bert-large-uncased           15360             16            0.089     
      bert-large-uncased           16384             16            0.091     
--------------------------------------------------------------------------------
TPU was used for training. Note that the time after compilation stabilized (after ~10 train loss=model.forward(...) + loss.backward() calls) was measured.

```

## Environment info

Running on GKE cluster, TPUv3-8, vanilla tpu-pytorch/xla:r1.5 image, XRT_TPU_CONFIG set

```
====================        ENVIRONMENT INFORMATION         ====================
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
- transformers_version: 3.0.0
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.5.0a0+6d48871
- python_version: 3.6.10
- system: Linux
- cpu: 
- architecture: 64bit
- date: 2020-07-01
- time: 14:37:15.608184
- fp16: False
- use_multiprocessing: False
- only_pretrain_model: False
- cpu_ram_mb: 30156
- use_gpu: False
- use_tpu: True
```
That looks like some solid batch parallelization :D Yeah these results don't look very accurate.

To be honest TPU Benchmarking is not very well tested yet and probably not very reliable, also partly because PyTorch/XLA is not very robust yet either. I will try to see if I can find the reason for this.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
