Could you try to use the nightly binaries with CUDA11.3 or CUDA10.2 (in case you are using an older GPU) via:
pip3 install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu113/torch_nightly.html