Hi there everyone, Pre-processing with BERT based models has been the biggest cause of pain for me while working on my project (Android), currently I am using sentencepiece’s C++ library (I was already using NDK for other things) and it works as I expected, kinda ! But I would like to figure out if there is a way to integrate it with the tflite model itself because it’s a lot of code removed if that could be done.
This is one of the Notebooks 2 I was experimenting on before doing it the C++ way, it takes less than 5 minutes to run the entire notebook (pretty small) and make the tflite embedding model and to reach the error when you run the tflite model.

interpreter.invoke()
RuntimeError                              Traceback (most recent call last)
 in ()
----> 1 interpreter.invoke()
/usr/local/lib/python3.7/dist-packages/tensorflow/lite/python/interpreter.py in invoke(self)
873     “”"
874     self._ensure_safe()
 → 875     self._interpreter.Invoke()
876
877   def reset_all_variables(self):
RuntimeError: Table not initialized.
(while executing ‘WordpieceTokenizeWithOffsets’ via Eager)Node number 2167 (TfLiteFlexDelegate) failed to invoke.

I can assume that there is some kind of table being used in pre-processing model which isn’t being initialized in the tflite version for the model when loaded into interpreter or during allocation. But I have no idea how to go about debugging it.