A. ENV

centos 7
Linux localhost.localdomain 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
gcc version 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC)
Python 3.8.5 (default, Sep 4 2020, 07:30:14)
libtorch 1.8.1 CPU version
x86_64 4CPU

B. TEST CODE
torch::jit::script::Module module;
module = torch::jit::load(argv[1]);
std::vectortorch::jit::IValue inputs;
at::Tensor input = torch::ones({1, 300, 40});
inputs.push_back(input);

for (int i=0; i < num; i++) {
    b = time_get_ms();
    std::cout << "module forward begin" << std::endl;
    at::Tensor output = module.forward(inputs).toTensor();
    e = time_get_ms();
    d = e - b;

    std::cout << "forward output size : " << output.numel() << ", No." << i <<" forward time : " << d << std::endl;

    usleep(100 * 1000);  // sleep 100ms
}
avg_duration = total_duration / num ;
std::cout << "AVG forward time : " << avg_duration << std::endl;

C. Performance


When I commented out the “usleep(100 * 1000)” code, I found that latency is good.


When I keep the “usleep(100 * 1000)” code, I find that latency is bad .


The above phenomenon is inevitable .


D. Question

Why do I get worse performance when I execute the “usleep(100 * 1000)” code ?
Especially in the multi-threaded scenario (for example. at::set_num_threads(4)), it is more serious ;
The performance data is in the comments.