Looking at the model def you posted, it looks like it is not yet quantized.  One missing thing is calibration. You can add a calibration step after you call prepare and before you call convert:
torch.quantization.prepare(fused_model, inplace=True)

# calibrate your model by feeding it example inputs
for inputs in your_dataset:        
    fused_model(inputs)

print('Quantized model Size:')
quantized = torch.quantization.convert(fused_model, inplace=False)
print_size_of_model(quantized)