I have a model that will use more memory than is available on the GPU

one option could be to train it on the CPU, but that sounds slow, though not impossible
one option could be to split the model across multiple GPUs, but I’d rather avoid that, for various reasons
an option that occurs to me is something like:

run part of the forward pass on one gpu
somehow move the intermediate results of that into main memory, clean them out of gpu memory (except the outputs themselves)
rinse and repeat
in the backward pass, calculate the gradients flowing back to these outputs
move each of the sets of intermediate results onto the gpu, and pass the relevant gradients through them
doable?
faster than just running everything on cpu???  (not faster in dev time I imagine…)



Thoughts?