we have had torch.distributed since August 2017.
See:
https://pytorch.org/docs/stable/distributed.html 1.4k
https://pytorch.org/docs/stable/nn.html#distributeddataparallel 287
https://pytorch.org/docs/stable/nn.html#distributeddataparallelcpu 1.6k