Solved by Vasiliy_Kuznetsov in post #2 
              
              
                Currently int8 kernels are not supported on GPU.  The workaround is to either eval on GPU without converting to int8 (with fake_quant emulating int8), or to eval int8 on CPU.  Adding int8 support on CUDA is being considered but there is no timeline at the moment.