@pelias interesting issue you are facing.
I just tried to install on a new conda env based on python 3.7 and could not repro the issue:
(test) user:/# conda install pytorch torchvision cudatoolkit=11.3 -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/test

  added / updated specs:
    - cudatoolkit=11.3
    - pytorch
    - torchvision


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cudatoolkit-11.3.1         |       h2bc3f7f_2       549.3 MB
    freetype-2.11.0            |       h70c0345_0         618 KB
    giflib-5.2.1               |       h7b6447c_0          78 KB
    intel-openmp-2021.4.0      |    h06a4308_3561         4.2 MB
    jpeg-9d                    |       h7f8727e_0         232 KB
    libidn2-2.3.2              |       h7f8727e_0          81 KB
    libwebp-1.2.0              |       h89dd481_0         493 KB
    lz4-c-1.9.3                |       h295c915_1         185 KB
    mkl-2021.4.0               |     h06a4308_640       142.6 MB
    mkl-service-2.4.0          |   py37h7f8727e_0          56 KB
    mkl_fft-1.3.1              |   py37hd3c417c_0         172 KB
    mkl_random-1.2.2           |   py37h51133e4_0         287 KB
    numpy-1.21.2               |   py37h20f2e39_0          23 KB
    numpy-base-1.21.2          |   py37h79a1101_0         4.8 MB
    olefile-0.46               |           py37_0          50 KB
    pillow-8.4.0               |   py37h5aabda8_0         644 KB
    pytorch-1.10.0             |py3.7_cuda11.3_cudnn8.2.0_0        1.21 GB  pytorch
    pytorch-mutex-1.0          |             cuda           3 KB  pytorch
    torchvision-0.11.1         |       py37_cu113        30.3 MB  pytorch
    typing_extensions-3.10.0.2 |     pyh06a4308_0          31 KB
    ------------------------------------------------------------
                                           Total:        1.92 GB

and
(test) user:/# conda install ignite -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/test

  added / updated specs:
    - ignite

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ignite-0.4.7               |             py_0         129 KB  pytorch
    ------------------------------------------------------------
                                           Total:         129 KB

The following NEW packages will be INSTALLED:

  ignite             pytorch/noarch::ignite-0.4.7-py_0

Finally, I have
(test) user:/# conda list | grep pytorch
ffmpeg                    4.3                  hf484d3e_0    pytorch
ignite                    0.4.7                      py_0    pytorch
pytorch                   1.10.0          py3.7_cuda11.3_cudnn8.2.0_0    pytorch
pytorch-mutex             1.0                        cuda    pytorch
torchvision               0.11.1               py37_cu113    pytorch

I wonder if it is not related to the usage of conda-forge channel …
PS @ptrblck thanks for pinging !