Hi, sorry for the late reply.
First regarding your question, I was seeing these 0.4s and 0.03s almost in every runs.
Later, i found that this issue can be fixed by adding the following sentence   at::init_num_threads();
Thanks again for your kind help!