In our scenario, there is only one process to serve the request, and the process will be closed only when an exception occurs. Each model inference algorithm is complied into a so file and loaded into this process.  Multiple model inference algorithms will be loaded in this process. Each so file contain a tensorflow static library, the relationship is described as flows.

image (1)1351×531 28.1 KB

In the service, the so file of different model inference algorithms (model _ 1.so…) will often be loaded and closed through dlopen and dlclose. Because some static variables in tensorflow are created through the “new” method , as shown below. When closing the “so” with dlclose，these static variables will not be released， which will cause memory leaks.

image21242×250 57.4 KB


image31190×366 173 KB


image41250×204 137 KB

What is the purpose of tensorflow designed like this？Will it be optimized for this problem in the future.