@mingruimingrui thanks for sharing. May I ask how the tokenizer behaves in case in some example cases below:

How can user mentions (e.g., @DemoUser)  or hashtags (e.g., #helloworld) tokenized
Does “This is…a test” get correctly tokenized?
Does "So much fun  Good night! get correctly tokenized (please assume no whitespaces before and after the emojis)
Does “Hi :-p” get correctly tokenized, i.e., the emoticon remains token “:-p”

I work a lot with social media, and missing whitespaces, emojis as sentence separator, etc. are very common. Most off-the-shelf tokenizer struggle with most cases. And even some that, for example recognize, common emoticons such as “:-)” struggle with varitions like “:-)))”.
This is why I had to bend over backwards to write my own tokenizer that covers most common cases, but I’m happy to try others.