Hi, since you mentioned that 2 processes with 1 GPU each attains the expected speedup in the Google cloud env, it leads me to think that there may be a difference in configuration between that and the Docker env. Could you try the following and see if it works for you?
Setting export OMP_NUM_THREADS=1 as explained in https://github.com/pytorch/pytorch/issues/22451 42
For your hypothesis about slower GPU to GPU communication, it may be worthwhile to debug which portions of the training are slower on the Docker instances. You can add instrumentation to determine which parts of training (initialization, data loading, forward, backward, etc) are slower than on Compute Engine. One way to do this would be to use the torch.cuda.Event.elapsed_time() API (https://pytorch.org/docs/stable/_modules/torch/cuda/streams.html#Event.elapsed_time 2) to record GPU computations (an example is available here: https://pytorch.org/docs/stable/notes/cuda.html#cuda-semantics 1)