Hi, world_size = 1 and rank = 0 wouldn’t really work for distributed training as we generally want to train with > 1 GPU (i.e world size > 1). In particular # of GPUs (not necessarily # of nodes) is usually used as the world size.
Regarding slowness in loss.backward(), can you provide a repro of that (it appears that your code snippet is an allreduce above)? In general, loss.backward() will trigger additional allreduces during the backwards pass to synchronize parameter gradients, but especially for 2 GPUs we don’t expect this to add significant overhead.