Hi Pranavan (and Jin)!



 Pranavan_Theivendira:

model.hidden1.weight[0, 0].requires_grad = False



No, this won’t do what you want.  requires_grad applies to entire
tensors and not individual elements.  And to prevent unexpected
errors, pytorch won’t let you do this.
Probably the easiest way to freeze part of a tensor is to store the
value of the element you want frozen, let the optimizer modify the
entire tensor, and then restore the original value of the element
you want frozen.
Thus:
>>> import torch
>>> torch.__version__
'1.9.0'
>>>
>>> _ = torch.manual_seed (2021)
>>>
>>> lin = torch.nn.Linear (2, 3)
>>> lin.weight[0, 0].requires_grad = False
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().
>>>
>>> opt = torch.optim.SGD (lin.parameters(), lr = 0.1)
>>>
>>> lin.weight
Parameter containing:
tensor([[-0.5226,  0.0189],
        [ 0.3430,  0.3053],
        [ 0.0997, -0.4734]], requires_grad=True)
>>> w00 = lin.weight[0, 0].item()
>>> lin (torch.ones (2)).sum().backward()
>>> opt.step()
>>> lin.weight
Parameter containing:
tensor([[-6.2264e-01, -8.1068e-02],
        [ 2.4303e-01,  2.0531e-01],
        [-3.3811e-04, -5.7337e-01]], requires_grad=True)
>>> with torch.no_grad():
...     lin.weight[0, 0] = w00
...
>>> lin.weight
Parameter containing:
tensor([[-5.2264e-01, -8.1068e-02],
        [ 2.4303e-01,  2.0531e-01],
        [-3.3811e-04, -5.7337e-01]], requires_grad=True)

Best.
K. Frank