What I mean is to move part of those intermdediate outputs to CPU RAM and only use them when backprogation reaching them.
For example,
z = f(x), y=g(z), after I get z, I move intermediate outputs of f(x) to CPU RAM and then start to compute g(z). Same idea for backpropagation but reversely.