Can you provide the model code which you are trying to quantize. FYI quantization is not implemented yet for CUDA
At the moment PyTorch doesn’t provide quantized operator implementations on CUDA - this is the direction for future work. Move the model to CPU in order to test the quantized functionality.