In my case, just using the pip option --no-cache-dir was sufficient to make the install work.

About my config, I have 1Mo of RAM on my AWS EC2 debian server. Here are the outputs of lsb_release and free, after the installation of torch:
admin@server:~$ lsb_release -cds
Debian GNU/Linux 10 (buster)
buster
admin@server:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:          987Mi       439Mi       246Mi        10Mi       301Mi       396Mi
Swap:            0B          0B          0B