To add to that,
I’m unsure about the goal here, pretty much the only reason to do QAT is to learn the best weights when weight quantization is applied. QAT’s central thesis is about how you can propagate gradients across a discontinuous function that quantizes teh weights, so doing QAT without weight quantization is a bit of a mismatch.
If you only want to quantize the activations, you may want to consider using something more suited to it.