Solved by taha_entesari in post #7 
              
              
                I just wanted to let everybody know what the problem was. Hoping that nobody makes this foolish mistake  
The issue was that I didn’t train my network. I mean, now that I load a pre-trained model and then quantize, the model output isn’t all zeros (I haven’t checked accuracy difference). 
I hadn’t i…