If you want to build PyTorch from source, follow these instructions 155.
Note that you don’t need a local CUDA installation for the binaries, as they will install cudatoolkit from a conda directly. You would just need the NVIDIA driver on your machine.