Hi @dskhudia, my model isn’t public, but you can recreate the error with just Bert base like this:
import torch
from transformers import BertConfig, BertForSequenceClassification

config = BertConfig.from_pretrained('bert-base-uncased')

model = BertForSequenceClassification.from_pretrained(
    'bert-base-uncased', config=config)

model.to('cpu')

quantized_model = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8)

quantized_model.save_pretrained('model_dir_q')

model = BertForSequenceClassification.from_pretrained('model_dir_q')