Thanks for the resource links, unfortunately my application usage is summarization @lgusm
Integrate BERT natural language classifier | TensorFlow Lite 7 is what made me do the current sentence piece through C++ Api, I went the manual way because in my brief flipping through source codes, it didn’t seem to have any public functions regarding just the pre-processing (if my memory serves correctly), Also, I am pretty sure they use out-of graph pre-processing just like my implementation anyways so, it seemed like I am not losing any performance improvements.
My main motive for me to start the thread was because I started to port my application to flutter, and I wanted to move the pre-processing complexity to model side. But that seems very unstable at the moment as saw @cpoohee’s story with my own tinkering (hope this recieves love from the tflite devs). Currently I am exploring the option of using dart ffi to wrap sentence piece’s C++ api as this way worked before I started moving to flutter, if I succeed (it seems very promising from today’s testings), I will try to publish this to pub.dev 1 so others can use it if they require.