This is probably due to your model definition and not related to pytorch itself. In general libtorch is only a C++ API to the backend which is also used by the Python API. I found it to be as fast as caffe (which makes sense, since the backends have been/are on process to/will be merged)