Thank you for responding. I think I got it now. Computational graph should be connected. (Any way you suggest to confirm that? would be a good check to make while designing any model)
So loss.backward() is solely responsible for calculating all the gradients of loss function in the computational graph wrt to the parameters and optimizer.step() just picks up these gradient values to make an update.
Sorry, I thought optimizer.step() also has role in deciding what all (and how) gradients will be computed (based on parameters passed to the optimizer).