For example, if you consider the relu operation for a 1D input of size n. The Jacobian would be an  n^2  matrix.
But what the backward pass does is just mask out some entries in  v  as:  v[inp<0] = 0 .

I see, but what if you have a convolution? Correct me if I’m wrong, but at that point, you would have to convolve the convolution output with the respective input region for taking the gradient w.r.t the weights right?
What if the weights are functions say of a tensor product of a smaller amount of parameters (Imagine a separable 2d convolution, where I decide to compute the kernel using the tensor product)? At that point, pytorch would first compute the gradient for each weight of the full 2d kernel, and then the result would be multiplied by the gradient of the tensor, which is similar to the (A * B) * C operation above, right?
Anyways, thank you for your time answering my question. If I see that the method above applies to my situation I definitely implement it.