Thank you so much for your time and helpful reply.

Also I am fairly confused by this line: param.grad = nn.Parameter(torch.from_numpy(newGrad.detach().numpy()).float()) why do you wrap the grad in a Parameter? And why do you send it to numpy?

I tried reading it up before replying. I was going through 11-785 Deep Learning 2 and I think PyTorch is similar. I am still confused by autograd (even after a week) if I have to manually apply gradients myself.
From what I know, Parameter wraps a numpy array as a tensor. Because, I want to select the gradients manually and apply at each iteration, my gradients are in numpy array (because I am more familiar with numpy as compared to PyTorch).

You will need to update the .grad fields before calling the optimizer.step() function I’m afraid.

Ok. But, how do I invoke autograd.grad() for a model (without applying gradients to the model)? I was seeing Automatic differentiation package - torch.autograd — PyTorch 1.8.1 documentation . So, do I do model.grad()?