Installing the newest PyTorch indeed did help, so that is great! Still, there seems to be significant (factor 2) slowdown if I run work in parallel, compared to running it serially.
I attach a reproducible example, where a lot of inference is done in parallel. (Note that the inference model is random here, to keep things simple. In production, this would be a pretrained model…) When the workers run in parallel, they take about 100 ms to complete. In series, they take about 45 ms. So, accounting for the fact that I run 16 cores, I get a 7* speedup from parallelization, where I would expect 14-16 times, since this is (or should be) completely CPU-bound. I believe this model is too small (and especially batch of 1) for PyTorch to do meaningful parallelization under the hood here.
Ps I hope I didn’t do any stupid things in implementation here. Any improvement to the implementation here is of course also very much appreciated!


  
      
      GitHub
  
  
    

WillemvJ/parallelInference 15

MWE parallel inference in pytorch c++ front-end. Contribute to WillemvJ/parallelInference development by creating an account on GitHub.