I try to run the script on 0.4.0 and it works fine with torch.autograd.profiler.profile(use_cuda=True).
It seems this problem should be solved on upgrading to 0.4.0.
import torch
cuda = torch.device('cuda')
x = torch.randn((1, 1), requires_grad=True)
print(x.device)
with torch.autograd.profiler.profile(use_cuda=True) as prof:
        y = x ** 2
        y.backward()
print(prof)