Solved by ptrblck in post #2 
              
              
                You shouldn’t recreate a tensor, as it would break the computation graph. Use torch.cat or torch.stack instead: 
p = nn.Parameter(torch.ones(1) * 2)
out = torch.cat([p**i for i in range(10)])
out.mean().backward()
print(p.grad)
> tensor([409.7000])