It is not necessary to convert at the end of every epoch. The fake quantized model is expected to be very close to the final converted model, so one conversion at the end should suffice. We are in the process of aligning the numerics better between fake quant and converted model so if you are seeing some differences they should get better.