Could you update to PyTorch 1.7.0 or the nightly release, as I get a proper error now:
RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

so it seems to be an already fixed issue.