Okay, so the saturation happens during multiplication, but not the accumulations during convolution.

The saturation I mentioned earlier, happens during multiplication. However, for large number of channels even the accumulations into an int32 might overflow. This should be very rare and should only happen for a very large number of channels. For such a case, there is no saturation but the accumulator overflows and wraps around.

Actually, I am confused that why using uint8 not int8.

We are using uint8 for activations because the x64 vector instruction to multiply two 8-bit integers (vpmaddubsw) accepts uint8 for one input and int8 for another. We chose uint8 for activations and int8 for weights. int8 for activation can also be supported by a preprocessing step, i.e., by converting int8 to uint8 and adjusting zero_point for activations (add 128 to all int8 elements and new_zero_point = old_zero_point + 128 in the preprocessing step).