Perhaps forward mode is used loosely in this literature; what I mean here is that, contrary to reverse mode, I don’t have to store the all instances of my weights in memory to compute the final hypergradient. It does not refer to forward mode on dLtrain/dparams.
My question is: in the snippet above, can this sort of operation (for loop + writing one col at a time) be turned into a single matrix multiplication and single call to autograd.grad?