I tried to remove pytorch from my anaconda environment with:
conda remove pytorch pytorch-cpu
then I reinstalled pytorch with the instructions from here:

  
      

      pytorch.org
  

  
    

PyTorch 3

  An open source machine learning framework that accelerates the path from research prototyping to production deployment.


  

  
    
    
  

  


which led me to the following command:
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
This is the output of this command which seems to be still installing pytorch with cpu and not for CUDA.
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.10.1
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /home/fabrizioschiano/anaconda3/envs/deeplabv3finetuning

  added / updated specs:
    - cudatoolkit=11.3
    - pytorch
    - torchaudio
    - torchvision


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cpuonly-1.0                |                0           2 KB  pytorch
    ninja-1.10.2               |   py37hd09550d_3         1.5 MB
    pytorch-1.7.0              |      py3.7_cpu_0        59.5 MB  pytorch
    torchaudio-0.7.0           |             py37         9.8 MB  pytorch
    torchvision-0.8.1          |         py37_cpu        17.2 MB  pytorch
    ------------------------------------------------------------
                                           Total:        88.0 MB

The following NEW packages will be INSTALLED:

  cpuonly            pytorch/noarch::cpuonly-1.0-0
  libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0
  ninja              pkgs/main/linux-64::ninja-1.10.2-py37hd09550d_3
  pytorch            pytorch/linux-64::pytorch-1.7.0-py3.7_cpu_0
  torchaudio         pytorch/linux-64::torchaudio-0.7.0-py37
  torchvision        pytorch/linux-64::torchvision-0.8.1-py37_cpu
  typing_extensions  pkgs/main/noarch::typing_extensions-3.10.0.2-pyh06a4308_0

The following packages will be UPDATED:

  cudatoolkit        conda-forge::cudatoolkit-10.1.243-h03~ --> pkgs/main::cudatoolkit-11.3.1-h2bc3f7f_2


Proceed ([y]/n)? y


Downloading and Extracting Packages
ninja-1.10.2         | 1.5 MB    | ################################################################################################################################################## | 100% 
cpuonly-1.0          | 2 KB      | ################################################################################################################################################## | 100% 
torchaudio-0.7.0     | 9.8 MB    | ################################################################################################################################################## | 100% 
torchvision-0.8.1    | 17.2 MB   | ################################################################################################################################################## | 100% 
pytorch-1.7.0        | 59.5 MB   | ################################################################################################################################################## | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: - By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html

done


Do you have any suggestions?