Thanks for the reply, I work on a project together with Jim. I have read trough the docs you linked, but I am not sure if I understand how the interop threads are used. To make things clear, we are creating a model procedurally inside c++, right now we are working on simple feed forward nets, where we feed linear modules of varying size and activation functions into a sequential, and use the forward method of sequential module to run inference. We are testing on different shapes of the network and different sizes of the data. We can see clear performance difference when setting different number of intraop threads, but basically none with different interop threads. The docs say “PyTorch uses a single thread pool for the inter-op parallelism, this thread pool is shared by all inference tasks that are forked within the application process.”, now the thing I struggle to understand is how does one “fork” a new thread that uses the interop threads form libtorch (I can see in pytorch example you use a _fork() function from jit, but I cant find anything similar in libtorch)? Or does the forking happen somehow automatically inside libtorch code? Another thing the docs say is "One or more inference threads execute a model’s forward pass on the given inputs. Each inference thread invokes a JIT interpreter that executes the ops of a model inline, one by one. ", again, sorry for my lack of understanding, what does the “one or more inference thread…” mean here, do I need to fork the inference threads my self or does libtorch do it for me? Also it says that that each inference thread invokes JIT interpreter, from my understanding JIT interpreter is used to run TorchScript models, but when I have a model inside c++ in form of a sequential module, is the JIT interpreter still being invoked?