Hi,
You can look at the DistributedDataParallel module to perform distributed training.
Otherwise, we very recently added to master support for autograd with the distributed package. (not the multiprocessing package, the distributed package).