Okay, so I created a new conda environment like this:
conda create -n dl1 python=3.6 
Then, I activated the environment created above and ran the command to install the latest version:
conda install pytorch torchvision cpuonly -c pytorch
After that, at the python prompt:
>>> import torchvision

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/iseadmin/.local/lib/python3.6/site-packages/torchvision/__init__.py", line 1, in <module>
    from torchvision import models
  File "/home/iseadmin/.local/lib/python3.6/site-packages/torchvision/models/__init__.py", line 11, in <module>
    from . import detection
  File "/home/iseadmin/.local/lib/python3.6/site-packages/torchvision/models/detection/__init__.py", line 1, in <module>
    from .faster_rcnn import *
  File "/home/iseadmin/.local/lib/python3.6/site-packages/torchvision/models/detection/faster_rcnn.py", line 7, in <module>
    from torchvision.ops import misc as misc_nn_ops
  File "/home/iseadmin/.local/lib/python3.6/site-packages/torchvision/ops/__init__.py", line 1, in <module>
    from .boxes import nms, box_iou
  File "/home/iseadmin/.local/lib/python3.6/site-packages/torchvision/ops/boxes.py", line 2, in <module>
    from torchvision import _C
ImportError: libcudart.so.10.0: cannot open shared object file: No such file or directory