I am finally able to achieve it by setting different config for the layers i want to quantize, rather than using model.qconfig (which does for all the layers in the model).
For example:
I quantized the first layer by accessing the first layer and assigning it the qconfig.
per_channel_quantized_model.features[0][0].qconfig=torch.quantization.get_default_qconfig(‘fbgemm’)
per_channel_quantized_model.features[0][0].qconfig=torch.quantization.get_default_qconfig(‘fbgemm’)
Hope this helps. And if there is any other efficient approach to achieve this please do let me know.