I think the “slowdown” is expected, as you are creating the CUDA context, which would load all kernels for the used compute capability. You could additionally check the memory usage via nvidia-smi (add a sleep into your code if needed) and in my setup I see ~255MB (without libtorch) vs. ~697MB (with libtorch).