That expectation is built into the torch.distributed.launch utility but not elsewhere. You can start 5 processes (1 per GPU) and use world_size=5 where you have 2 processes on one machine and 3 processes on the other machine. It’s not very common to have this situation, so I’m not surprised most of the examples you see assume a symmetric contribution across machines. That said, you can still make it work, but will have to adapt those examples or start from scratch with torch.nn.parallel.DistributedDataParallel.