Maria_Vazhaeparambil:

countor_net = torch.quantization.convert(countor_net, inplace=True)
countor_net.cuda()
countor_net.eval()



Hi @Maria_Vazhaeparambil , this snippet is the part which is not supported.  When you do torch.quantization.convert, the fp32 kernels get swapped to int8 kernels.  There is currently no support to run int8 kernels on the GPU.  If you’d like to evaluate a model with int8 kernels, it has to be done on CPU, so you would need to move your converted model to CPU.