I am trying to implement the GradNorm from https://github.com/hosseinshn/GradNorm/blob/master/GradNormv8.ipynb 9 wherein I am using a pretrained word2vec embedding layer for which I have set requires_grad=False.
On cell 4 of the Notebook there is
G1R = torch.autograd.grad(l1, param[0], retain_graph=True, create_graph=True).
The above line throws the following error
G1R = torch.autograd.grad(l1, param[0], retain_graph=True, create_graph=True) File "/usr/lib64/python3.6/site-packages/torch/autograd/__init__.py", line 145, in grad inputs, allow_unused) RuntimeError: One of the differentiated Tensors does not require grad.
Is there anyway to implement this code for my usecase where pretrained embedding can have requires_grad=False ?