Pure cpu cluster is also supported by distributed package. See the doc for torch.distributed for details. You probably can’t use the built-in DDP, but you can use the primitives in the distributed package to write your own training code.