gphilip:

What would have been the outcome if Autograd did not know how to do this? An exception/error message saying something about gradients not being computable?


Yes, Autograd would yield an error, if the backward method is not available.
This would happen e.g. if you are using a non-differentiable operation:
x = torch.randn(10, 10, requires_grad=True)
out = torch.argmax(x, dim=1)
out.backward(torch.ones_like(out))
> RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn




 gphilip:

I cannot figure this out: what is the derivative of with respect to merely rearranging data? I am not sure if there is a way to define this. But perhaps since there is no change in the values we can take this derivative to be zero? Is this what Autograd is doing here? It is adding the zeros to the original ones, so the ones stay where they are?


No, squeeze and unsqueeze do not change any values and the number of elements of the tensor stays the same so the gradient is just passed to the tensor.