In Model parallelism, A DNN is divided into sub-modules and each module is handled by a GPU.
The forward graph, I assume, spans multiple GPUs.
Will the backward graph along with any internal data also span multiple GPUs?