Hi, I believe -quantization-precision only supports Int8 and Int16 right now. For 4-bit quantization, it’s currently only supported by a few ops, such as EmbeddingBag and SparseLengthsSum given that these ops often are loading from extremely large embedding tables that can be shrunk significantly using 4-bit quantization.
If you wanted to use it for other operators we’d need to expand its support across a variety of different operators.