It’s very, very slow compared to the roughly equivalent nested loop pure C++ code below the torch loop in the example I posted. And the nested loop stuff doesn’t even use something like Blas which I think would potentially be even faster.
In general I don’t understand pytorch. I want to use a neural network in my C++ application. I get the weights from somewhere else, so no training is involved. I just end up running a bunch of forward calls for inference similar to the example I posted above. Doing this with libtorch is very slow. Even just creating a tensor for the forward input without even running forward slows the code down at least an order of magnitude.
I tried doing the same thing with an equivalent network using a random C++ NN implementation  I found here:

  
      
      GitHub
  
  
    

yixuan/MiniDNN 12

A header-only C++ library for deep neural networks - yixuan/MiniDNN


  
  
    
    
  
  


and the code runs something like 750x faster than libtorch. Is this not a good use case for libtorch? Am I missing something?
At this point I’ve tried building from source and also running on a different cluster. Neither changed anything.