BramVanroy:

rather than to have manually install a new venv with PyTorch and manually setting the correct paths for CUDA


If you don’t need to rebuild from source and would be alright using the stable release of the nightly binaries, you wouldn’t have to worry about CUDA at all, since the conda binaries and pip wheels ship with the CUDA, cudnn, NCCL etc. runtime libs.
You would only have to make sure your workstation has the right NVIDIA driver installed.
If you are often rebuilding PyTorch and need a proper CUDA setup, I would recommend to try out the pytorch docker containers (and rebuild inside them) so that you won’t have to figure out where to install which package etc.