As an alternative to manually navigating to rename Blender’s site-packages/numpy directory I found it can also be explicitly removed via pip.
As another reference; these were the commands I used to install pytorch on Windows:
python -m ensurepip
python -m pip install --user --upgrade pip setuptools wheel
python -m pip uninstall -y numpy
python -m pip install --user --upgrade https://download.pytorch.org/whl/cu90/torch-1.0.1-cp37-cp37m-win_amd64.whl
python -m pip install --user --upgrade torchvision

These were the commands I used to install pytorch on Linux:
./python3.7m -m ensurepip
./python3.7m -m pip install --user --upgrade pip setuptools wheel
./python3.7m -m pip uninstall -y numpy
./python3.7m -m pip install --user https://download.pytorch.org/whl/cpu/torch-1.0.1.post2-cp37-cp37m-linux_x86_64.whl
./python3.7m -m pip install --user torchvision

(In both cases making sure to run the python binary that’s distributed with Blender itself)
I checked the instructions here 8 to confirm the appropriate pytorch wheel url for Linux/Windows and Python 3.7 (matching the version distributed with Blender).
It hardly matters but I chose to generally install everything outside of Blender’s site-packages via --user just to minimize my modifications to the based installation of Blender.
Only tested with Blender 2.8