Hi,
I’m currently working on a single-GPU system with limited GPU memory where multiple torch models are offered as “services” that run in separate python processes.
Ideally, I would like to be able to free the GPU memory for each model on demand without killing their respective python process. However, if I only delete the models (and empty the cache) without killing the process, I’m still stuck with the memory usage of the cuda context, which is about 1.2 GB per process in my case.
In general, the memory usage of the context seems extremely high to me, especially when considering that the model itself only adds about 200MB to it while it’s loaded.
So my question is: Are there any options to decrease the memory footprint of the cuda context or to get rid of it altogether when running inference on the models?
I’d be grateful for advice!