Hi,
We cannot track gradients across processes indeed with the regular autograd.
If you really want this, you can take a look at the experiemental distributed autograd we built on top of rpc here 48. Note that this is experimental so there might be some rough edges.