Solved by ptrblck in post #6 
              
              
                Yes, this should work as seen here: 
x = nn.Parameter(torch.randn(2, 2))
y = x**2

y.backward(torch.ones_like(y) * 2)
print(x.grad)
print(2*x*2)