HUSTHY:

Could not run ‘quantized::layer_norm’ with arguments from the ‘CPU’ backend. 'quantized::layer_norm’ is only available for these backends: [QuantizedCPU]


Hi @HUSTHY, this error message means that a fp32 tensor is being passed into the quantized layernorm kernel.  There are at least two options to address this:

add a torch.quantization.QuantStub() to convert the tensor to int8 before it enters the quantized layernorm layer.  For example,

# init
...
self.quant = torch.quantization.QuantStub()
self.layer_norm = torch.nn.LayerNorm(...)
...

# forward
...
x = self.quant(x) # convert a fp32 tensor to int8
x = self.layer_norm(x)
...


leave the layernorm in fp32.  This can be done by setting qconfig = None to prevent a layer from getting quantized.