makslevental:

maybe I could speed up a forward pass (at inference time) by partitioning up one of the layers (a very “tall” Conv2d - i.e. >20 output channels) across all of my GPUs. So I used DDP to map across my GPUs


Hey @makslevental could you please elaborate on how did you manage to use DDP after splitting one layer? Does it mean each DDP process now sees a different model? Or is it true that each DDP process no longer has exclusive access to its own GPU?

and surprisingly (or maybe not?) forward pass actually gets slower with increasing number of GPUs. Is this to be expected?

Since with the split, each forward pass will do cross-device communications, so it is possible to see slowdowns.