Thanks for your reply!



 mcarilli:

Any CPU thread in the process may submit work to any cuda stream (the kernel launch and stream API are thread safe), and the work may run concurrently with work submitted from other CPU threads.


One thing I don’t understand then, is how come in practice this example 17 can parallelize training (on single GPU), without MPS, and yet it doesn’t even use different cuda streams?
Does it mean it just happens on high level due to better utilization of “idle” times, where the GPU is not busy, and these times now can be filled by commands from other threads/processes? If that’s the case it’s surprising that the improvement is almost linear in the number of processes.



 mcarilli:

By default (without MPS) each device runs kernels from only one context (process) at a time. If several processes target the same device, their kernels can’t run concurrently and GPU context switches between processes will occur.


So it means that without MPS, work submitted from different CPU processes cannot run concurrently on the same GPU, but work submitted from different threads can (via streams)?
If I understand correctly, the example code above uses processes?