Hi @ptrblck.
I finally got it working!!. I did some extra steps. Not sure if it was needed though but I set a couple of env variables pointing to where I had cuda and cudnn libraries:
export CUDA_HOME="/usr/local/cuda-10.2"
export CUDA_NVCC_EXECUTABLE="/usr/local/cuda-10.2/bin/nvcc"
export CUDNN_INCLUDE_PATH="/usr/local/cuda-10.2/include/"
export CUDNN_LIBRARY_PATH="/usr/local/cuda-10.2/lib64/"
export LIBRARY_PATH="/usr/local/cuda-10.2/lib64"
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
export USE_CUDA=1 USE_CUDNN=1 USE_MKLDNN=1

After building I got these prints:
print(torch.__version__) # => 1.6.0a0+76c964d
print(torch.version.cuda) # => 10.2
print(torch.backends.cudnn.version()) # => 7605

Then I ran into an issue since I’m using also torchvision package. Installing it wasn’t an option since it was destroying my freshly compiled pytorch. Searching through the forum I ended up in this discussion, Error when building pytorch from source 30, where someone had similar issue. You actually suggested building also torchvision from source.
After all this, I no longer have this problem and finally I can use my old GPU .
Thanks a lot for your help.