Thank you for the extra data point.
looking at the already published wheels:  https://download.pytorch.org/whl/torch_stable.html 5
I see the following:
cu102/torch-1.7.0-cp37-cp37m-linux_x86_64.whl -> torch/lib/libtorch_cuda.so: 1.1GB
cu110/torch-1.7.0%2Bcu110-cp37-cp37m-linux_x86_64.whl -> torch/lib/libtorch_cuda.so: 2.2GB
So something caused this artifact to double in size between cuda 10.2 and 11.
And from what I can tell between cuda releases 4, it may just be cuda itself.
I also tried using CXXFLAGS/CFLAGS with "-Oz" to optimize size. but this seems to have had no effect.
Here are some of the other options we use during local build:
export EXTRA_CAFFE2_CMAKE_FLAGS=("-DATEN_NO_TEST=ON")
export TORCH_NVCC_FLAGS="-Xfatbin -compress-all"
export TORCH_CUDA_ARCH_LIST="6.1;7.5+PTX"
export TH_BINARY_BUILD=1
export CMAKE_LIBRARY_PATH="/opt/intel/lib:/lib"
export CMAKE_INCLUDE_PATH="/opt/intel/include"
export DEBUG=0
export CXXFLAGS="-Oz"
export CFLAGS="-Oz"
export USE_CUDA=1
export USE_NNPACK=0
export USE_QNNPACK=0
export USE_FBGEMM=1
export USE_OPENMP=0
export USE_TBB=1
export USE_GFLAGS=0
export USE_GLOG=0
export BLAS=MKL
export ATEN_THREADING=TBB
export MKL_THREADING=TBB
export MKLDNN_CPU_RUNTIME=TBB
export PARALLEL_BACKEND=NATIVE_TBB
export CC=/clang_9.0.0/bin/clang
export CXX=/clang_9.0.0/bin/clang++
export PATCHELF_BIN=/usr/local/bin/patchelf
export VERBOSE=1

Maybe “fatbin” is a problem.  Going to try that next.