I have a model that fits perfectly fine on any one of my GPUs (4x1080Tis) but I had the bright idea that maybe I could speed up a forward pass (at inference time) by partitioning up one of the layers (a very “tall” Conv2d - i.e. >20 output channels) across all of my GPUs. So I used DDP to map across my GPUs and surprisingly (or maybe not?) forward pass actually gets slower with increasing number of GPUs. Is this to be expected?
I’m not an expert on the execution pipeline on GPUs but is it the case that any individual CUDA kernel (e.g. my “tall” Conv2d) gets executed in parallel? I’m guessing that that’s my issue - that the layer I’m partitioning up already gets executed in parallel and the scatter/gather just adds copy (and process instantiation) latencies.