mcarilli:

Without MPS, I don’t see how this example could run work from different processes concurrently


Not only that it works concurrently (in the sense that running the example with 10 processes vs. 1 results in much less than 10x slowdown), but in this case, MPS almost doesn’t help further.
I think I’ve convinced myself by now that I know how to start and stop an MPS server. I modified this code to use a simpler network (MLP) and not use a dataloader but instead just have all data in memory, and in this case MPS showed some improvement, but still far from linear. That’s how I knew that MPS works.
But okay, I guess this doesn’t explain much without profiling like you said. Maybe this example is too complex. Maybe there are idle GPU times implied in it which allow the speedup without MPS, and don’t allow MPS to help because the Cuda computations aren’t actually possible to parallelize due to low utilization.
That’s why it would be very helpful to see some very minimal Pytorch example that can demonstrate good MPS usage, i.e. if you set num_processes=1, then the run takes X seconds, and then you change it to num_processes=10, and the runtime stays about X seconds (plus a small overhead), but without MPS it would be 10X.
Maybe you could provide a clue on how to write such minimal example? i.e, should I use torch.multiprocessing as in the example above? or something else for dispatching processes to the GPU?