I’m interested in parallel training of multiple instances of a neural network model, on a single GPU. Of course this is only relevant for small models which on their own, don’t utilize the GPU well enough.
According to this 23, Pytorch’s multiprocessing package allows to parallelize CUDA code.
I think that with slight modification of this example code 33, I managed to do what I wanted (train several models) instead of training a single model using the Hogwild algorithm, and it worked pretty well. I managed to train 10 models in less then 150% the time it took to train a single one.
Is this functionality related in any way to Nvidia’s MPS 29 capability? I suspect not, because it doesn’t seems to involve anything related to starting an MPS server from the system, setting the GPU to work in EXCLUSIVE mode and so on.
I understand that MPS is the way in which Nvidia supports CUDA multithreading/multiprocessing. Otherwise, it’s not really possible to tell the GPU to do things in parallel.
Is this true? if it is then what exactly multiprocessing does and how does it work so well on the GPU without using MPS? 
Can anyone point me to an example which does demonstrate the use of MPS with Pytorch? I couldn’t really find one. And strangely enough, the above 28-page Nvidia guide on MPS doesn’t include any example in Pytorch or any other leading framework.
Thanks!