This is because there is duplicated QuantStub placed in the model, I think forward should be:
def forward(self, x):
    # manually specify where tensors will be converted from floating
    # point to quantized in the quantized model
    x = self.quant(x)
    x = self.features(x)
    x = self.avgpool(x)
    x = torch.flatten(x, 1)
    x = self.classifier(x)
    # manually specify where tensors will be converted from floating
    # point to quantized in the quantized model
    x = self.dequant(x)
    return x

If you set qconfig for the full model, then all modules will be quantized, the output of self.classifier will be quantized as well. Why do you call self.quant(x) in the end instead of self.dequant(x) in the end?