alinak:

So my question is: Are there any options to decrease the memory footprint of the cuda context or to get rid of it altogether when running inference on the models?


You could reduce the CUDA context size by removing kernels e.g. via dropping libraries such as MAGMA or cuDNN, if these are not used.
Besides that I’m not aware of reducing the operator set in the framework itself.