eqy:

What is the amount of time spent loading data vs. doing computation on each of the GPUs?


I’m not sure exactly, judging based on nvidia-smi I’d estimate 50% time spent loading and 50% doing computation (this estimate is based on % of time I saw each GPU being utilized).



 eqy:

Additionally, the use of torch.cuda.empty_cache() can add overhead if it is done for every batch.


You’re right it is probably not necessary it is just a remnant of a previous solution I tried to deal with a memory leak (which is now gone).
I will retest without clear cache and get back to you.