Bigger batch size decrease the throughput.
I'm training bert-base-uncased from scratch, single node, 8xA100, if increase the batch size, the throughput decrease, detail is:

- set batch size to 256, the throughput is 7000 sample/s, GPU Utilization rate 80%
- set batch size to 384(while full gpu memory), the throughput is 4000 sample/s, GPU Utilization rate 30%

It look like the IO problem(or maybe not, you can correct me)?
Any help will be appreciated~ txs
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.