Hi,
Yes in theory it does, but in practice, since none of these matrices are actually materialized, you might not get the speedup you would think by just looking at the matrix products.
In particular the convolution backward is a transposed convolution which might be tricky to adapt to do what you want to do