Thanks for discussion @Nikronic. Beside thread swith, the time to memory io may also affect the performance for small tensors because the io time is relatively took more cost in total processing time.
And I did a test to divide the inputs [1, 32, 100000] to 8 part, each part is [1, 32, 12500]. And runs conv for each part in 8 thread. Compare to running in single thread, this can speed up about 4x. The only additional thing to do is concat each threads’ result and deal with the boundaries. Seems pytorch (and mkl-dnn backend) is not doing small conv with big inputs in this way.