You pretty much can’t do quantized inference with cuda, there are no native quantized cuda kernels atm, our team is working to support lowering to custom backends using Fx to TRT but its not complete yet.
also maybe @jerryzh168 can confirm, but I believe the intended solution was to do:
goal: set int_repr of a quantized tensor x to 3.
# x is per channel quantized tensor
x_int = x.int_repr()
x_int[0][0][0][0]=3
x_new = torch._make_per_tensor_quantized_tensor(x_int, x.per_channel_scales(), ... )

not
x.int_repr()=3
which i’m fairly sure is not intended to work