Oh I see, thanks!
Regarding the speed-up. It is probably less than for data parallelism, right? Because for module 2 to compute the forward pass, you need to know the results of module 1. (And vice versa for backprop.)
So the advantage would be though that it would allow for fitting basically large models into memory rather than training speed?