conda install torchvision -c pytorch

Hm, I thought if you would select the CPU version, it would not be automatically be installed.

Is there any possibility to only install torchvision without those two packages?

not sure about that, but
either
conda install ... --no-dependencies

or
conda install ... --no-update-dependencies

may work.

I guess CUDA Toolkit won’t be installed on the conda environment, but system-wide

No, it will be installed only for your conda environment. By default, if you are not using a conda virtual environment, the path would be ~
~/miniconda3/pkgs