Hi @yichong96
To clarify, does the first time you evaluate (after running convert) work fine?
I would try to inspect the model parameters and state_dict using both approaches (i.e. directly running eval, vs re-creating the quantized model and loading weights).
If you have a minimal example that could repro this issue, we are happy to take a look.