Does CUDA 9.0 have a similar issue? I seem to have the same problem.
I’m using 1080Ti, CUDA 9.0, CUDNN 7003. I’m installing PyTorch from source initially, but when I tried to install torchvision  using conda, my PyTorch installation might be overridden by conda. When I try conda list, I can see both
torch                     0.2.0+59e0472             <pip>
pytorch                   0.2.0           py36hf0d2509_4cu75    soumith

Running a simple script like this gives me the following results:
import torch
from datetime import datetime

for i in range(10):
    x = torch.randn(3, 4)
    t1 = datetime.now()
    x.cuda()
    print(i, datetime.now() - t1)

0 0:06:24.108245
1 0:00:00.000110
2 0:00:00.000055
3 0:00:00.000048
4 0:00:00.000046
5 0:00:00.000046
6 0:00:00.000044
7 0:00:00.000044
8 0:00:00.000044
9 0:00:00.000044

Sorry that I’m new to PyTorch so I might be doing something incorrectly. Thanks in advance.