However, I was wondering why it is expressed as a decimal number (float) instead of an integer (int) when logging.

For the same reason you don’t output a bunch of 1’s and 0’s when you print an fp32 number. All these numbers are just 1’s and 0’s in your computer. Its what they represent that’s important. You never print our an fp32 number, you print out the decimal representation of the 1’s and 0’s. The ‘non integer’ values that are being output for qint8 are the same, those are the actual values that it represents. In order to do efficient computations with these values, it utilizes some aspects of int8 data storage/ops, but thats only really important if you are trying to mess with things at a really low level. For most purposes its better to just think of qint8 as similar to fp16 i.e. not limited to integers, takes up less space, has lower fidelity compared to fp32.


When saving a model in Dynamic Quantization, does PyTorch Quant lib save it as int and convert it to fp32 when loading? (If yes, why?)


They are stored as qtensors (qint8) which, as mentioned before, is different from int8. pytorch/linear.py at 3e43c478a8832cec063aa566583a05f87d7dc3b0 · pytorch/pytorch · GitHub This is the deserialization code which takes the weight and bias and then packs them into the format that is necessary for the efficient computation of Linear.


Does Dynamic Quantization not use Quantized_operators optimized for int operations?


Yes it does generally use int8 operations (in part) when performing quantized operations. Here is a reasonable explanation of how qint8 Linear can be broken down into other ops, including int8 ops, to speed things up: Quantization for Neural Networks - Lei Mao's Log Book 1 (the first part explain quantization at a reasonable level, the part I’m talking about though is the Quantized Matrix Multiplication section)