I’ve just tested this on GPU and the CUDA function also has the exact same problem.
This is the setup:

PyTorch Version (e.g., 1.0): 1.3.1
OS (e.g., Linux): Linux
How you installed PyTorch (conda, pip, source): conda
Build command you used (if compiling from source): N/A
Python version: 3.6.9
CUDA/cuDNN version: 9.1
GPU models and configuration: Tesla V100
Any other relevant information: N/A