Oh yeah! It worked! And I’ve now learnt that pip wheels ship with their own CUDA runtime!
Installing torch with CUDA 11.1 with the following command did fix the initial issue with torch 1.8:
 pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

May you have an amazing day! Good luck @shamoons with you issue, do try out @ptrblck’s suggestions and hopefully you’ll be set!