Hi,
I am recently working with one huge model.
I am really curious can we split this model into several submodules and we process it with one single GPU in multi-stage?
For example, we iteratively put one submodule on GPU and forward, then we fetch it back and put another submodule on GPU.
Will backward work with this pipeline? I guess it works for forward mode.
Thanks.