OK, so I fixed the issue. I had to add some more options before building. I used this:
export USE_OPENCV=1 TORCH_CUDA_ARCH_LIST="5.1;6.1;7.5" 

I also added NCCL and MPI support
It was probably the CUDA_ARCH_LIST that was the key, i guess.
Anyway, it now works with both 1070 Ti and 2080 Ti