Hi,
Let’s say a layer has input X, and produces output Y.
In the original form, we need to store at each layer a weight matrix W of size m*m.
In the compact form, we need to store at each layer a weight matrix U of size (m/2)*(m/2), and on-the-fly generate a deflated version W = U[B]. We use W to generate the layer output Y, and W & B can be discarded. Back propagation only need to retain U & Y at each layer on the graph.
Not sure if my understanding of autograd is correct, or is there a way to get this behavior?
Thanks!