Yes! That worked! Everything went well with it. Pytorch was installed, and is working with cuda. Thank you so much peterjc123!
For future readers, to sum up:

I reinstalled drivers for CUDA, reinstalled CUDA with VS extensions and Nsight Compute (actually with everything checked), reinstaled CUDnn
went with

pip install ninja
set CMAKE_GENERATOR=Ninja
set CC=cl.exe
set CXX=cl.exe
set CUDAHOSTCXX=
set DISTUTILS_USE_SDK=1
for /f "usebackq tokens=*" %i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -version [15^,16^) -products * -latest -property installationPath`) do call "%i\VC\Auxiliary\Build\vcvarsall.bat" x64 -vcvars_ver=14.11
python setup.py build --cmake
python setup.py install