I tried this and got params.grad as None before loss.backward() and Zero after backward() step in the first iteration. If I continue for the next iterations, params.grad as zero both before and after as well, I think this is because in the first iteration grad is already zero and they are not updating.
Actually, to give a bit of an idea about architecture, I am trying to use the model to predict the parameters of the other network(let say InferenceModel()). The model architecture has 60 neural networks, each of which has 3 (inp, hid, out) layers. These 60 neural networks predicts(regress) 60 different parameters of InferenceModel(). This inference model takes input as images and outputs images(which means the loss is L2 between ground truth image and inferred image). So with this loss, I backpropagate the model's parameters. So I backpropogate on the loss computed using the output of InferenceModel() to update model parameters.
Model architecture is shown below for 1st two networks out of 60. The same NN blocks will be repeated for remaininnig 58 NN as well.
(1): FCBlock(
      (net): Sequential(
        (0): FCLayer(
          (net): Sequential(
            (0): Linear(in_features=512, out_features=128, bias=True)
            (1): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
        (1): FCLayer(
          (net): Sequential(
            (0): Linear(in_features=128, out_features=128, bias=True)
            (1): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
        (2): FCLayer(
          (net): Sequential(
            (0): Linear(in_features=128, out_features=262144, bias=True)
            (1): Tanh()
          )
        )
      )
    )
    (2): FCBlock(
      (net): Sequential(
        (0): FCLayer(
          (net): Sequential(
            (0): Linear(in_features=512, out_features=128, bias=True)
            (1): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
        (1): FCLayer(
          (net): Sequential(
            (0): Linear(in_features=128, out_features=128, bias=True)
            (1): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
        (2): FCLayer(
          (net): Sequential(
            (0): Linear(in_features=128, out_features=512, bias=True)
            (1): Tanh()
          )
        )
      )
    )

FCBlock code
FCLayer code