Yes, but I’d like gradients of loss2 with respect to the original parameters (i.e… dloss2_by_dnetweight and not dloss_by_dnetweight).
To be clear, there’s no problem if torch.nn.functional is used with manually defined weights like this:
import torch
from torch import nn
from torch.nn import functional as F

w = torch.tensor([[1., 2.]], requires_grad=True)

x = torch.tensor([[3., 2.]])
y = torch.tensor([[2.]])

f = F.linear(x, w)
loss = (f - y)**2

grads = torch.autograd.grad(loss, w, create_graph=True)

w2 = w - 0.01*grads[0]
f2 = F.linear(x, w2)
loss2 = (f2 - y)**2

loss2.backward()

print w.grad # This works as expected.

My problem is with propagating gradients through nn.Parameter of an nn.Module.