part of the challenge when you run on Desktop / Server is not so much the size of PyTorch, but the size of depedencies that make deployment also efficient.
On x86 CPU, you will want Intel MKL and MKLDNN (which PyTorch install provides) which themselves are ~70MB and ~50+MB respectively. They are used for fast matrix multiplications and neural network operations.
on GPU, you will want CUDA and CuDNN, which take about 2GB