One possibility would be to remove specific libraries, such as NCCL, cudnn etc., which should reduce the context, as less code has to be stored.
Alternatively, you could try to remove PyTorch CUDA kernels, but this sounds pretty painful and you could be running in a lot of failures for undefined functions.