Sorry no, at each step I use reverse mode for the gradients of the training loss w.r.t model params, and I use that to compute Z. However I use forward mode manually by using the Z at the final step, to get the final (hyper)gradient of interest. So Z is an intermediate quantity that I need to compute the final hypergradient at step T, which is dL_val/dhyperparams =  (dLval/dparams_T) * (dparams_T/dhyperparams) = (dLval/dparams) * Z_T
If you’re familiar with MAML in meta learning, this is like using reverse mode for the inner loop, and forward mode for the outer loop