Hi @moskomule,
I keep getting (none,none)  as an hvps. What is autograd.grad() supposed to return?  Following is the (slightly modified) code:
from torch import autograd
linear = nn.Linear(10, 20)
x = torch.randn(1, 10)
out = linear(x).sum()
grads = autograd.grad(out, linear.parameters(), create_graph=True)
flatten = torch.cat([g.reshape(-1) for g in grads if g is not None])
x = torch.randn_like(flatten)
print(flatten.shape, flatten.requires_grad)
p=(flatten @ x)
p.requires_grad=True
hvps = autograd.grad(p,linear.parameters(), allow_unused=True) **
print(hvps)
output is:
===>
torch.Size([220]) False
(None, None)
**  without allow_unused=True, I get err: One of the differentiated Tensors appears to not have been used in the graph. Any idea what this means: which graph is it referring to?
I don’t understand the solution; why adding relu helps here? Can you help?
Thanks,