By adding cudatoolkit-dev to the list of installed packages I got a proper nvcc in my environment. Now I moved on to the next problem:
File /path_to_conda/miniconda3/envs/pytorch_build/lib64/stubs/libcuda.so doesn't exist

So now I’m trying to figure out how to get this file. It looks like there’s still something wrong/missing with my cuda-installation. Is magma-cuda101 the relevant package or what am I looking for?