shivamsaboo17:

n your example if x requires grad and I change the value by slicing (for example x[0, :] = torch.zeros((5))) , is it still differentiable? x requires grad but zeros tensor does not in my case.


The answer here is “it depends”, and it is a bit subtle.

The good news is yes. We can doa = torch.randn(5,5, requires_grad=True)
b = a+1
b[0] = torch.zeros(5)
b.tanh().sum().backward()

and it’ll work!
The bad news is there are a number of caveats:

You cannot do this on a leaf tensor (e.g. a above) or you get an error that the leaf was pulled into the interior of the graph. So you may need to .clone() the leaf. That said, if you need it for leaves, doing the zeroing in with torch.no_grad(): and manipulating a.grad after the backward might be a more efficient alternative.
This works in the above example because the original b is not needed to compute the backward. If you replace b = a + 1 with b = a.tanh() you will get the famous one of the variables needed for gradient computation has been modified by an inplace operation error message because a.tanh() actually wants to use the result to compute the derivative (which is 1-result**2). Again, you could do b = a.tanh().clone() to avoid this.



An alternative (but also with copying) is to not modify the things inplace that require grad, but set up a new tensor (without requires grad) and copy in the relevant bits.
b = torch.zeros_like(a)
b[1:] = a[1:]

This also works when copying (non-overlapping) bits from several tensors.
To me, the autograd handling of subtensor assignments is one of the most intriguing - but also subtle - things in autograd.
Best regards
Thomas