Hi all,
I read about dynamic quantization and wanted to replicate it in our environment. I actually got pretty good results… when using a single thread. However, we have a multithreaded GRPC application and I did not see any improvement when using multiple threads. In fact with 10 threads, the original (non-quantized) model actually had better throughput, although it was also using 5x CPU.
So I started to wonder if locking is to blame, specifically python’s GIL. Profiling our code revealed that it spends by far the majority of its time in linear_dynamic function which is defined here:  https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qlinear_dynamic.cpp 1
I noticed that code in python_torch_functions.cpp makes an explicit call to release the GIL:
pybind11::gil_scoped_release no_gil;
But I don’t see the same call in qlinear_dynamic.cpp
I tried to add this myself but I am having difficulty compiling torch from source. It’s much more complicated than  the documentation makes it out to be. I also don’t have enough information to determine whether apply_dynamic_impl() method is thread-safe.  Can any pytorch engineers comment? There is potentially a massive performance bottleneck here.
thanks,
Eugene