You probably don’t need to downgrade the CUDA 11 installed in your system. As explained here 1.1k,  conda install pytorch torchvision cudatoolkit=10.2 -c pytorch  will install CUDA 10.2 and cudnn binaries within the Conda environment, so the system-installed CUDA 11 will not be used at all.
I recently installed ubuntu 20.04 and Nvidia driver 450. It took me a while to realize that I didn’t have to build pytorch from source just because I have CUDA 11 in my system.