Using eager mode quantization often changes the model hierarchy of the model, so it’s possible that the module hierarchy of your quantized model no longer matches your fp32 state_dict. What is the origin of the fp32 state_dict, is it also from torchvision?
In practice, people usually fix this by either loading the weights before fusing the model, or by writing custom state dict mappers which modify the state keys according to the module hierarchy changes.