Have you checked the accuracy of fused_model? By checking the accuracy of fused_model before converting to int8 model we can know if the issue is in the preprocessing part or in the quantized model.
If fused_model has good accuracy, the next step we can check the quantization error of the weights. Could you try the following code:
def l2_error(ref_tensor, new_tensor):
    """Compute the l2 error between two tensors.

    Args:
        ref_tensor (numpy array): Reference tensor.
        new_tensor (numpy array): New tensor to compare with.

    Returns:
        abs_error: l2 error
        relative_error: relative l2 error
    """
    assert (
        ref_tensor.shape == new_tensor.shape
    ), "The shape between two tensors is different"

    diff = new_tensor - ref_tensor
    abs_error = np.linalg.norm(diff)
    ref_norm = np.linalg.norm(ref_tensor)
    if ref_norm == 0:
        if np.allclose(ref_tensor, new_tensor):
            relative_error = 0
        else:
            relative_error = np.inf
    else:
        relative_error = np.linalg.norm(diff) / ref_norm
    return abs_error, relative_error

float_model_dbg = fused_model
qmodel_dbg = quantized

for key in float_model_dbg.state_dict().keys():
    float_w = float_model_dbg.state_dict()[key]
    qkey = key
    
    # Get rid of extra hiearchy of the fused Conv in float model
    if key.endswith('.weight'):
        qkey = key[:-9] + key[-7:] 

    if qkey in qmodel_dbg.state_dict():
        q_w = qmodel_dbg.state_dict()[qkey]
        if q_w.dtype == torch.float:
            abs_error, relative_error = l2_error(float_w.numpy(), q_w.detach().numpy())
        else:
            abs_error, relative_error = l2_error(float_w.numpy(), q_w.dequantize().numpy())
        print(key, ', abs error = ', abs_error, ", relative error = ", relative_error)

It should print out the quantization error for each Conv weight such as:
features.0.0.weight , abs error =  0.21341866 , relative error =  0.01703797
features.3.squeeze.0.weight , abs error =  0.095942035 , relative error =  0.012483358
features.3.expand1x1.0.weight , abs error =  0.071949296 , relative error =  0.010309489
features.3.expand3x3.0.weight , abs error =  0.18284422 , relative error =  0.025256516
features.4.squeeze.0.weight , abs error =  0.088713735 , relative error =  0.011313644
features.4.expand1x1.0.weight , abs error =  0.0780085 , relative error =  0.0126931975
...