Hi,
No we don’t actually try to do that.
The main reason is that we almost never actually do such vector matrix product.
The backward pass implementation is given v and computes v^T Jf with Jf being the Jacobian of the current function f.
So we don’t actually need to materialize Jf ever.
For example, if you consider the relu operation for a 1D input of size n. The Jacobian would be an n^2 matrix.
But what the backward pass does is just mask out some entries in v as: v[inp<0] = 0.