In the end I switched from Conda to virtualenv and it worked at the first try.
I created my virtualenv with virtualenv virtualenv_name
Then I did
workon virtualenv_name
then, I installed pytorch as it is specified on the official pytorch website (but selecting pip instead of conda) as package manager (Start Locally | PyTorch 36).
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
This worked for me and now I have a CUDA-enabled version of pytorch on my machine.