For QNNPACK, the time you measure is on a x86 CPU. I am not surer how fast QNNPACK quantized implementation is compared to float on x86.
Can you try with fbgemm on a desktop to see what you get?
Also,  you should not reuse the same module (self.add.add) multiple times in forward. This is because each instance needs to collect its own statistics.  This will lead to poor accuracy with the quantized model.