The first CUDA operation on a device will create the CUDA context (which contains the PyTorch kernels, cudnn, NCCL etc.). While you are able to delete all tensors, you are not able to release the CUDA context during the script lifetime.