PyTorch quantization supports int8 (but not int4), with fast kernels for CPU on mobile via QNNPACK.  https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html 30 has some information to get started, and you would want to set the backend to qnnpack to target mobile CPUs.