@ptrblck Hi, thank you for your kind reply. In my case, I need to calculate the Hessian matrix of loss w.r.t the weights of model for each data point (using torch.autograd.grad()). There is no need update weights. May I ask do I need to explicitly clone the models for each processes to make sure the gradients will not be shared?