Would this also work for one 1 gpu with two sequential steps somehow? If my model is too large to fit on one gpu can I somehow do the forward/backward pass sequtially where I only have one part in gpu memory and somehow cache the other part for the backward pass later.
Somehow like this:
x = submodule1(x)
#somehow unload intermediate results of submodule1 from gpu here and cache for later backward pass 
#(and then load on gpu again when needed in backward pass of submodule1)
x = submodule2(x)

I could imagine how this works but then I don’t know how I would pass the gradients that come from submodule2 back to submodule1 and initiate the backward pass on submodule1.