You can use the model.layer.qconfig = None syntax to turn off quantization for a layer and all of its children.  Please feel free to see https://pytorch.org/docs/stable/quantization.html#model-preparation-for-quantization 1 for more context.