So right now I don’t think we have automatic Glow-based quantization support for this. We have pre-quantized Glow kernels for executing these ops, but they are only ever loaded pre-quantized from the input model, i.e. they are quantized in PyTorch, Caffe2, etc. before Glow ever loads them.
In order to support this we’d need to extend the Glow profiler to (1) support per-row profiling, and then (2) use that per-row profiling to do the 4-bit quantization.
(Note that our 4bit quantization support for EmbeddingBag and SparseLengthsSum are both rowwise quantized)