Other information is that I was building the python module using pybind11 2.4 with C++ standard 11. I have seen some related question on pybind11 thread that talked about potential memory leak using C++ standard 11 building pybind11 2.4.