Solved by Vasiliy_Kuznetsov in post #2 
              
              
                Hi @HUSTHY, this error message means that a fp32 tensor is being passed into the quantized layernorm kernel.  There are at least two options to address this: 

add a torch.quantization.QuantStub() to convert the tensor to int8 before it enters the quantized layernorm layer.  For example,

# init
..…