This is because torch quantized concat op just piggy back to FP32 by
dequnatize all inputs -> do concat in FP32 -> quantize concatenated tensor

So it will never be faster than FP32 concat. See the implementation

  
      github.com
  
  
    pytorch/pytorch/blob/f6c46df856d0588360db5b807960d1fc5e888c36/aten/src/ATen/native/quantized/cpu/qconcat.cpp#L61-L66 5

  xs.push_back(qx.dequantize());
}
const Tensor y = at::cat(xs, dim);
Tensor qy;
AT_DISPATCH_QINT_TYPES(x_dtype, "qcat", [&]() {
  qy = at::quantize_per_tensor(y, scale, zero_point, SCALAR_TYPE);



  
  
    
    
  
  


There are other operators that follow the same FP32 fallback approach (hence slower than FP32) such as quantized elemwise add, mul etc.