I would assume this is expected, since the FakeQuantize uses some additional operations on the tensor values to fake the quantization.

PyTorch 1.3 doesn’t provide quantized operator implementations on CUDA yet - this is direction of future work. Move the model to CPU in order to test the quantized functionality.
Quantization-aware training (through  FakeQuantize  63) supports both CPU and CUDA.