I must admit I didn’t use virtualenv but just sudo pip3’d it into localhost on Debian unstable. I try to avoid having numpy reinstalled, so I pass --no-deps, but the system pip3 liked the torch packages and put everything neatly under /usr/local/lib/python3.5.
Best regards
Thomas