Our source build with TORCH_CUDA_ARCH_LIST="5.2 6.0 6.1 7.0 7.5 8.0 8.6+PTX" creates a libtorch_cuda.so of 1269MB with CUDA11.1, so I’m unsure why your build creates a bigger lib with less compute capabilities.
We are currently working internally to provide CUDA and cudnn conda and pip wheels, which would allow to dynamically link these runtimes for the binaries. However, this would not solve your local build issue.