XeL:

Good news! I was able to compile it using TensorFlow Lite. I’ll test it out, but as @Igusm point it out, it weight 278 MB, so I guess I’ll have trouble using it on the web.


Off the top of my head - have you tried any of the quantization techniques for model size reduction mentioned in in the TensorFlow Lite docs? I hope some of the following stuff helps:


Model optimization  |  TensorFlow Lite 6

Post-training quantization  |  TensorFlow Lite 4
Post-training dynamic range quantization  |  TensorFlow Lite 7
Post-training integer quantization  |  TensorFlow Lite
Post-training float16 quantization  |  TensorFlow Lite 1
Post-training integer quantization with int16 activations 1
TensorFlow Lite 8-bit quantization specification



Also, in case you haven’t checked this out - there are TensorFlow Lite Model Maker guides and tutorials specifically for NLP (QA and classification) (cc @billy):


TensorFlow Lite Model Maker 1

BERT Question Answer with TensorFlow Lite Model Maker
Text classification with TensorFlow Lite Model Maker 1



And, if you are into ML research:


Google AI Blog: Advancing NLP with Efficient Projection-Based Model Architectures 1

pQRNN is quantized, further reducing the model size by a factor of 4x.

Link to arXiv:


[1712.05877] Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (Jacob et al., 2017 - Google)



Quantization Aware Training with TensorFlow Model Optimization Toolkit - Performance with Accuracy — The TensorFlow Blog (2020, TensorFlow blog)