Solved by ptrblck in post #2 
              
              
                You could reduce the CUDA context size by removing kernels e.g. via dropping libraries such as MAGMA or cuDNN, if these are not used. 
Besides that I’m not aware of reducing the operator set in the framework itself.