In FBGEMM, we have requantization as a post-processing step to convert the 32 bit to 8 bit after the accumulation. The requantization basically does the following op in the language of numpy:
X_q = (np.round(X / X_scale) + X_zp).clip(0, 255).astype(np.uint8)

The modularized requantization wrapper is here:

  
      github.com
  
  
    pytorch/FBGEMM/blob/master/include/fbgemm/OutputProcessing-inl.h#L59-L185 4

template <
    bool FUSE_RELU,
    QuantizationGranularity Q_GRAN,
    typename BIAS_TYPE,
    typename outT,
    typename inT,
    typename nextOPType>
template <inst_set_t instSet>
inline int
ReQuantizeOutput<FUSE_RELU, Q_GRAN, BIAS_TYPE, outT, inT, nextOPType>::f(
    outT* out,
    const inT* inp,
    const block_type_t& block,
    int ld_out,
    int ld_in) const {
  static_assert(
      std::is_same<inT, int32_t>::value,
      "input data type must be of int32_t type");
  int ncol_per_group = ncols_ / groups_;
  assert(


  This file has been truncated. show original

  
  
    
    
  
  


And inside it a efficient AVX2 kernel is implemented:

  
      github.com
  
  
    pytorch/FBGEMM/blob/master/src/QuantUtilsAvx2.cc#L533-L554 3

/*
 * Convert scaled FP32 result to int32_t using CVTPS2DQ instruction.
 * CVTPS2DQ instruction rounds result according to nearest FP32 value
 * with ties to even (assuming default MXCSR rounding mode). However,
 * when conversion overflows, it produces INT32_MIN as a result. For
 * large positive inputs the result of conversion can become negative,
 * which affects the final requantization result. Note that on x86
 * SSE2 we have e.g. int32_t(float(INT32_MAX)) == INT32_MIN! This
 * happens because float(INT32_MAX) rounds to 2**31, which overflows
 * int32_t when it is converted back to integer.
 *
 * Thankfully, we can prove that overflow never happens in this
 * requantization scheme. The largest positive input is INT32_MAX
 * (2**31 - 1), which turns into 2**31 when converted to float. The
 * largest scale value is 0x1.FFFFFEp-1. When multiplied together, the
 * result is 2147483520 (compare to INT32_MAX = 2147483647), which
 * fits into int32_t without overflow.
 */
__m256i x_rounded_v = _mm256_cvtps_epi32(x_scaled_v);
__m256i y_rounded_v = _mm256_cvtps_epi32(y_scaled_v);


  This file has been truncated. show original