Solved by ptrblck in post #4 
              
              
                Yes, you are right and don’t need to install a local CUDA toolkit (just the drivers) unless you want to build PyTorch from source or a custom CUDA extension.