Solved by tom in post #4 
              
              
                The answer here is “it depends”, and it is a bit subtle. 

The good news is yes. We can doa = torch.randn(5,5, requires_grad=True)
b = a+1
b[0] = torch.zeros(5)
b.tanh().sum().backward()

and it’ll work!
The bad news is there are a number of caveats:

You cannot do this on a leaf tensor (e.g. a abo…