Solved by supriyar in post #2 
              
              
                Hi @huoge - for int8 dynamic quantization we quantize the weights to 8-bits so you see the expected size reduction. 
For fp16 quantization, the weight values are cast to fp16 (taking saturation into account), but the dtype is still set to float32. This has to do with the type expected by the FBGEMM …