Solved by masahi in post #4 
              
              
                This is because torch quantized concat op just piggy back to FP32 by 
dequnatize all inputs -> do concat in FP32 -> quantize concatenated tensor

So it will never be faster than FP32 concat. See the implementation 
 
 
There are other operators that follow the same FP32 fallback approach (hence slow…