Ok, for the benefit of those who might hit this issue, it is being tracked here: https://github.com/pytorch/glow/issues/2836 8.
To summarize: essentially the CPU backend kernels, which is libjit in backends/CPU/libjit has to be compiled for the correct architecture. This can be achieved by adding appropriate flags in backends/CPU/CMakeLIsts.txt for the CPURuntimeCompilationOptions. The sources in libjit may have to be modified/customized for the specific architecture (e.g. some includes may need to be changed or removed).