Hello ptrblck. Thank you for your reply.
I understand that your advice is quite resonable, but I have to say that I’m using the optimizer as provided in Pytorch and also GCP as it is.
While I could search for the bottlenecks as you suggested, I wonder if the Pytorch platform and the sample program are designed to fully take advantage of multi-GPU settings. So, I should restate my question as follows.
Do the sample program and Pytorch codes linearly scale with batch sizes when run on multiple GPUs?
If the codes scale with no problem when used on some multi-GPU hardware, then I suppose that the multi-GPU settings of GCP might have a problem with Pytorch.
Maybe I misudetstood something, but I just wondered if it is a common practice to care about the GPU botttolnecks even when using Pytorch. Thank you.