You could rebuild PyTorch and remove libraries shipping with device code, such as cuDNN. While this would yield a performance hit, no cuDNN kernels will be loaded and stored in the CUDA context.
Yes, you can use PYTORCH_NO_CUDA_MEMORY_CACHING=1 to disable the cache.
PyTorch needs to load its own kernels as well as device code from other libs (cuDNN, cublas, NCCL, etc.), which might not be the case for PyCUDA. It might be possible to share some driver-related code in the context, but I don’t know how much memory savings would be expected and haven’t experimented with it.