Thanks for posting @dyukha, I am seeing that you are running the code inside a jupyter notebook, I think the error you hit here is not a problem with PyTorch distributed, this is related to the incompatibility between python multiprocessing module and Jupyter Notebook because multiprocessing module pickles data to send to processes. If you can try the script in a normal python file it should work fine.