The GPU allocation is not unreasonable since one of the 4 GPUs has to store all optimizer state since the gradient updates are always done on only one of the four GPUs.
I have never seen such a heavy slow down when using multiple GPUs. Note also though, that if you leave per_device_train_batch_size=16 when doing multi GPU training you increase your effective batch size by a number of 4 in your setup. But I’m not really sure what’s going on here…what happens if you reduce your batch_size?
Finally, it is recommend by PyTorch to do distributed training instead of “multi-GPU” training. Here is an example on how to do distributed training: transformers/examples at master · huggingface/transformers · GitHub 67