What is the amount of time spent loading data vs. doing computation on each of the GPUs?
Additionally, the use of torch.cuda.empty_cache() can add overhead if it is done for every batch. Is there some kind of issue where other processes are using the same GPU? Otherwise, this seems like it would be unnecessary.