If you aren’t using any linear algebra routines that would be provided by MAGMA, compiling with USE_MAGMA=0 should also give you around 100MiB? or so of memory savings.
Of course, once you are willing to pay recompilation costs and are willing to specialize your build for certain use cases, the world is your oyster when it comes to deleting unused GPU kernels to save memory usage…
The hard limit for this would probably be about 300MiB or the CUDA context size.