Solved by tom in post #2 
              
              
                As you observe, in many circumstances, a.grad will not require gradient. 
This is, however, not universally true: 
a= torch.randn((5,5), requires_grad=True)
(a**2).sum().backward(create_graph=True)
print(a.grad.requires_grad)

Note that .detach() is universally better than .data, except when you use…