This might be expected because the gradient will be for a “quantized approximation” of the current parameters rather than the parameters.
But so one thing you could do is first train a bit in FP32 and then do some QAT training.
You should not really lose much by looking in fp32 for where to start the QAT.
Best regards
Thomas