agreed with @ptrblck. @ClaartjeBarkhof  You can manually send the data across gpus with the communication ops provide by the distributed package, in this case you might consider using allreduce 1, scatter or reduce_scatter. If you also want to automatically support backward for those data communications, consider using the torch.distributed.nn to do those communications