I met this error again, and I found  the true reason is your param.grad and new grad aren’t in the same device.
so the best method to solve this error is
param.grad = torch.from_numpy(np.zeros(param.grad.shape)).type(param.grad.dtype).to(param.device)