Hi,
This is exactly what I mention above as “To get a small gain, you would need to use multiple processes which are not supported by autograd out of the box and so you will need to unpack/pack Tensors when sending them to make sure it will work.” in my answer above: you won’t be able to do multiprocessing with Tensors that are involved in autograd out of the box.
You will need to unpack/repack then and do the missing part of the autograd yourself.
Note that we now have a distributed-aware version of the autograd here: Distributed RPC Framework — PyTorch 1.7.1 documentation 28 but it is not as full-featured as the vanilla autograd.