You can have a large explanation here:
https://pytorch.org/docs/master/notes/cuda.html 1
The main short is ops aren’t processed in the order you call them but in an optimal one as internally ops are enqueued.
Some ops force synchronization.
That’s what i mean by “they are async”.
WRT using outputs as inputs u can do that. It will construct a graph and backprop properly despite they are not hosted by the same nn.Module. Autograd works at a tensor level.
If you just want to set a daemon process (which means, a thread which runs in the back) I don’t think you can do that in a trainable way. And, once again, even if you could do so, it won’t be faster than “sequentially”. If a proper training pipeline is designed, gpu usage should be ~100%. Thus, splitting in daemon processes ends up taking the same time.
Lastly, you can set 3 nn modules, 3 optimizers and 3 of everything if you need to.