alexgo:

One thing I don’t understand then, is how come in practice this example can parallelize training (on single GPU), without MPS, and yet it doesn’t even use different cuda streams?


Without MPS, I don’t see how this example could run work from different processes concurrently.  It’s possible the model is so small, and each process’s individual utilization is so low, that you observe a speedup with multiple processes even if they aren’t running concurrently.  Hard to tell without profiling 5.

work submitted from different CPU processes cannot run concurrently on the same GPU, but work submitted from different threads can (via streams)?

Streams are free-floating from threads.  Any thread may submit work to any stream.

If all threads submit work to the same stream, those kernels will be serialized, even if the threads are in the same process.  However, no context switches will be required.
If threads in the same process submit work to different streams, the kernels may run with true concurrency (overlapping).
If threads in different processes submit work to the same device, without MPS the kernels may not run concurrently regardless what streams are used, because they are in separate contexts.  MPS allows processes to submit work to the device without a context switch.  I believe (but i’m not sure) kernels may run with true concurrrency (ie, overlapping) even if processes each use their default stream.  But even if the kernels from different processes can’t truly overlap, MPS increasing the density of work submission by avoiding expensive context switches is beneficial.

Pytorch does not compile to use per-thread default streams.  By default, all threads submit work to a shared default stream (called “legacy” default stream in some cuda docs).  Therefore, unless you manually set stream contexts 5 in each thread, case 1 applies.