UPDATE : In the documentation 1 it’s wrote At the moment PyTorch doesn’t provide quantized operator implementations on CUDA - this is the direction for future work. Move the model to CPU in order to test the quantized functionality.
It’s may be the reason why.