Fine-tuning translation model speed anomalies
### System Info

python 3.8.12
ubuntu18.04
transformers 4.23.1
![image](https://user-images.githubusercontent.com/49591435/198196407-2a20a98e-130e-4699-97a9-63836cb231dd.png)


### Who can help?

@Narsil
@patil-suraj
@sgugger, @patil-suraj

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1FwXCjUVvrNpCuf0KoxhxQKTciRscva6t?usp=sharing

t4 graphics card half precision about 10, a100 is more than 300, even if not up to 30 times the speed should not be almost the same speed, and here the multi-threaded seems to be bad

### Expected behavior

After experiments found that bs has a greater impact on the speed, gradient accumulation also has a slight impact, but the same bs, a100 and t4 speed is almost the same, the expected is to start multi-threaded or other methods to accelerate, otherwise, training once to more than 100 days
Please use the [forums](https://discuss.huggingface.co/) to debug training like this as we keep features for bugs (with a clear reproducer) and feature requests only.<|||||>I think it is a bug in itself, I tried many devices and the speed is almost the same, obviously not reasonable<|||||>> t4 graphics card half precision about 10, a100 is more than 300, even if not up to 30 times the speed should not be almost the  same speed, and here the multi-threaded seems to be bad

I read this 3 times, and I still don't understand. What do you mean ?

Just as a note for here or the forums, trying to be over explicit might help readers understand what you're trying to do and what are your expectations.<|||||>Sorry, the message is machine translation, I mean there are two problems, the first problem is that different gpu speed should be different, a100 is better than t4, especially the semi-precision gap is big, so turn on the semi-precision training speed should be a big gap, however the speed is almost the same. The second problem is that multi-threading does not seem to have any accelerating effect, and can be improved a lot on cv, I do not know if this is the case in the nlp field
<|||||>8 card speed is no different from 1 card<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.