The CUDA context loads the driver and all linked CUDA kernels, i.e. PyTorch native kernels, cuDNN, NCCL, etc. If you don’t want to load these kernels (and either drop the performance or remove specific utils), you could rebuild PyTorch from source without any additional libraries (i.e. NCCL, cuDNN, MAGMA,…).