Solved by ptrblck in post #4 
              
              
                You can specify multiple GPU architectures by building with: 
TORCH_CUDA_ARCH_LIST="3.5 6.1" python setup.py install

where you can pass multiple architectures to the env var (as well as +PTX).