mcarilli:


If all threads submit work to the same stream, those kernels will be serialized, even if the threads are in the same process. However, no context switches will be required.
If threads in the same process submit work to different streams, the kernels may run with true concurrency (overlapping).
If threads in different processes submit work to the same device, without MPS the kernels may not run concurrently regardless what streams are used, because they are in separate contexts. MPS allows processes to submit work to the device without a context switch. I believe (but i’m not sure) kernels may run with true concurrrency (ie, overlapping) even if processes each use their default stream. But even if the kernels from different processes can’t truly overlap, MPS increasing the density of work submission by avoiding expensive context switches is beneficial.



That’s interesting. So according to this (case 3) I understand that this example may not be optimal for execution with MPS. It might be beneficial to try and further modify this example, and make sure that each process does it’s Cuda work on a different stream? because I don’t think this is the case now. I will try it out. Thanks!