No, it doesn’t. I can’t understand what’s the problem. Are you expecting a larger speedup? You’ve doubled the amount of computing power along with the input size, so the time staying constant is a great result.
Maybe you have some conditional branching in your model?

our multi-GPU code is nearly as fast as it can be. You still need the GIL to execute most of the autograd operations so it doesn’t matter if we don’t use Python threads. In our CNN benchmarks we have perf very similar to that of Caffe2, and faster than Lua Torch, which has very good multi-GPU implementation.