For add you could use torch.nn.FloatFunctional, the extra dequant and quant ops in the network could be slowing things down.
Regarding performance you can try running the torch.autograd.profiler on your model for some iterations to see which ops take up most time. It will give you an op level breakdown with runtime so you can compare float vs quantized model.