This error is usually raised, if the application wasn’t compiled for your GPU architecture.
I don’t know how PyTorch was built in your setup, but you might need to rebuild it either locally on the node with the desired GPU (which should detect the compute capability) or by specifying it manually via TORCH_CUDA_ARCH_LIST=6.1.