Hi, all.
I’m measuring wall clock time of executing a Cifar-10 sample program from Pytorch. I run the program on a virtual machine instance of GCP (Google Cloud Platform) with eight K80 GPUs. I’m testing Adam optimizer with various batch sizes.
I believe that I can increase the batch size as much as the GPUs can accept in the memories and that the GPUs perform multiplications at once within almost the same amount of time no matter how large the batch size is. This means that the one step of optimization for a batch size of 2048 takes almost the same time as 1024, resulting in halving execution time par epoch for 2048 compared to 1024.
However, one step of 2048 takes longer than 1024, and the execution time per epoch is almost the same in my experiment. I don’t understand the result.
I believe that reducing execution time by increasing the batch size is the only merit to have many GPUs. However, I haven’t had this merit in my experiment. The result was almost the same with batch size of 4096 and 8192, meaning that the execution time per epoch does not improve at all.
I understand that reducing time to reach target accuracy is important, but this is not relevant. I believe that GPUs are supposed to do many multiplications at once and are not supposed to increase the execution time as the number of multiplications increases. Would anybody please explain why this is not true in my case? Thank you in advance.