To clarify, are you wondering if you can store modules on RAM, and then load them as you need to? For example, if a two layer model is too massive, storing layer1 on the GPU and layer2 on ram?
Not really is the short answer. The modules need to be on the GPU to allow for the GPU to perform the forward and backward pass. You could do a thing where once the forward pass is completed on layer1, move it to RAM and move layer2 to the GPU, and continue the forward pass. But this will be quite slow to move between memory every forward pass (and I’m not sure how happy autograd will be about this).
However, you can just keep part of the model on the CPU. 2. It won’t be as fast as having it all on a GPU, but it’s a possible route.