amrmartini:

I know that similar issues have been brought up before. But I want to add that, it seems like this reduce_range parameter was introduced to resolve these issues. Yet, reducing the range of quantization by one bit will reduce accuracy, so reduce_range is not an all-round fix.


The fbgemm kernels require reduce_range to be True, to prevent potential overflow.  One could write a different kernel without this requirement, at the cost of reduced performance.
You are right that using one less bit is detrimental to accuracy.  There are a couple of strategies to improve accuracy - using moving average observers, using per-channel weight observers instead of per tensor, etc.