I’ve done a speed comparison between simple operators in libtorch and simple vectors. I don’t know entirely whether the comparison is fair, but the vector method seems to be about 60 times faster. Is there a way to improve speed on simple operators? I want to know whether it would make sense to convert back and forth from tensor to vector and back during inference on the CPU.
#include <torch/torch.h>
    Tensor X = torch::randn({ 253 });
    Tensor output = X;

    std::chrono::time_point<std::chrono::system_clock> time = chrono::system_clock::now();
    {
        torch::NoGradGuard no_grad;
        for (int i = 0; i < 10000000; i++) {
            output = X * X;
        }
    }

    cout<<chrono::duration_cast<chrono::milliseconds>(chrono::system_clock::now() - time).count()/1000.<<endl;
    //Convert tensor to vector
    std::vector<float> v(X.data_ptr<float>(), X.data_ptr<float>() + X.numel());
    vector<float> results = v;

    time = chrono::system_clock::now();
    for(int i=0;i<10000000;i++){

        std::transform(v.cbegin(), v.cend(), v.cbegin(), results.begin(), std::multiplies<float>());
    }
    cout<<chrono::duration_cast<chrono::milliseconds>(chrono::system_clock::now() - time).count()/1000.<<endl;

Libtorch: 7.125 seconds
Standard vector: 0.282 seconds