I did some more testing. So the program I use has inferencing on relatively small vectors on all threads. Above problems are with 1.10 (stable).
When I use PyTorch LTS (1.8.2), then the problems go away: When I spawn a single thread, PyTorch keeps at a single thread, i.e. it doesn’t spawn additional threads beyond the one created by me. As a consequence, with multithreading I get a huge speed-up.
Anyhow, for my use case it would be very helpfull if PyTorch 1.10 gains a switch to really turn off multithreading.