Hello!  I’m trying to do dynamic quantization as described here 10.
To quantize my own fine-tuned Bert model, I do this:
model = BertForSequenceClassification.from_pretrained('model_dir')
model.to('cpu')
quantized_model = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8)
quantized_model.save_pretrained('model_dir_q')

But when I later load the model, I get this error:
  File "/.../bert.py", line 329, in main
    model = BertForSequenceClassification.from_pretrained(args.output_dir)
  File "/.../transformers/modeling_utils.py", line 486, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
	While copying the parameter named "bert.encoder.layer.0.attention.self.query.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]).
	While copying the parameter named "bert.encoder.layer.0.attention.self.key.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]).
...

Would appreciate any guidance for how to fix this.