cuDNN is a library to accelerate workloads, such as convolutions etc., and will be used if it’s available, while CUDA is used to execute code on the GPU.
The binaries ship with both while you would need to install cuDNN separately into your local CUDA toolkit if you want to build PyTorch from source to be able to use it.