I just wanted to let everybody know what the problem was. Hoping that nobody makes this foolish mistake 
The issue was that I didn’t train my network. I mean, now that I load a pre-trained model and then quantize, the model output isn’t all zeros (I haven’t checked accuracy difference).
I hadn’t initially used a pre-trained model because since I had to change the underlying network a bit, I didn’t bother to modify the loading of state_dict and was just initializing the model randomly. I originally thought that this wouldn’t cause a problem because I thought PyTorch could quantize the activations well enough. But it seems that since the output is so random and not organized, the observer for the activations doesn’t work that well (the weights were correctly quantized as I had checked that the SNR of layers was good).