I have found that I was misconceptioned about quantized values as soon as byte value shows quant of its specific domain. I used the following paper as a guide https://arxiv.org/pdf/1712.05877.pdf 2 (though it’s a little superior to pytorch’s implementation because it provides Integer-Arithmetic-Only quantization, why many operations in pytorch quantization stay float).
With that I was able to successfully reimplement convolution and fully connected layers, but nn.quantized.FloatFunctional().add() still resists to be implemented. I I thought it would be the easiest layer, heh.
At the moment  tryed  the following reasoning:
S_3(q_3 - Z_3) = S_1(q_1 - Z_1) + S_2(q_2 - Z_2)

from where
q_3 = [S_1(q_1 - Z_1) + S_2(q_2 - Z_2)] / S_3 + Z_3

In that case almost half of the elements appear to be the same as in pytorch, but other half can change drastically. A lot of misses come from values needed to be clipped.

def manual_addition(x1, x2, add_layer):
#add_layer is of the type of nn.quantized.FloatFunctional()
q1 = x1.int_repr().numpy()
q2 = x2.int_repr().numpy()
z1 = x1.q_zero_point()
z2 = x2.q_zero_point()
s1 = x1.q_scale()
s2 = x2.q_scale()
z3 = add_layer.zero_point
s3 = add_layer.scale
q3_1 = s1 * (q1 - z1)
q3_2 = s2 * (q2 - z2)

qres = q3_1 + q3_2
qres = qres / s3 + z3
q3_int32 = (qres).round()

q3 = q3_int32.clip(0.255).astype(np.uint8) # many misses are from clipped values
gt_res = add_layer.add(x1, x2)
gt_res_int = gt_res.int_repr().numpy()
calc_hit_rate(q3, gt_res_int) #hit rate =0.51


If anyone has any clue on how clipping should be done here, I would appretiate your help so much.