QAT accuracy could depend on many things, including when do you turn on fake_quant and how many batches you train the model, are you training from scratch or from pretrained weights?
Also could you paste the modified model here?