If you install the binaries (via conda or pip) they will ship with their own CUDA runtime and you would only need to install the corresponding NVIDIA driver on your system.
Have a look here 1 (Table 1) to find the appropriate driver versions for different CUDA releases.