Solved by marvosyntactical in post #2 
              
              
                I found this quantization tutorial; the InvertedResidual class is basically what I want. So I will just change my model definition instead of modifying model code.