it could also be related to op support in QNNPACK.  PyTorch has a fork of QNNPACK which lives here (https://github.com/pytorch/pytorch/tree/172f31171a3395cc299044e06a9665fec676ddd6/aten/src/ATen/native/quantized/cpu/qnnpack 8), and the readme contains the supported ops.
Your model has a few modules which are not supported, which means they would still run but there aren’t fast ARM kernels: AdaptiveAvgPool2d, and Dropout.  Just for debugging’s sake, you could check if removing these modules or replacing them with alternatives which are optimized for ARM fixes the speed issue