Hi,
Is there a specific scheduling capability where I can define forward pass and backward pass to run on specific GPU devices? With model parallelism, it is clear that we can schedule the layers to be in specific GPU devices. But could we go further into the details? Without overriding, the backward function is it possible to give a config and train a neural network? This could involve a weight copy or activation copy to a separate GPU device to do the backward. Is this something possible with Autograd definitions and GPU scheduling in Pytorch?