Thanks for the detailed explanation @cpoohee I appreciate the time you took to explain the method, I got the gist of it from your last issue on how to go about doing it but this makes it much clearer. I am still a noob-ish in terms of depth of knowledge I have in ML.
Going to flutter makes it much more difficult (if i think to use sentence piece C++ lib) as now I have to go from dart to java (probably using a method channel) then from java to C++ using JNI bridge (I kinda have some of this setup but it will require more code), look like I may be able to skip some of it with dart ffi but I got no experience with it.
But for now I will keep the pre-processing and model separated as it seems a lot of work to integrate it right now.