Understanding GPT tokenizers
For those who interested, there some new researches in the field [0]. It usually possible to create more compact token representation from given text, but my guess the greedy "optimal" tokenizer might harm the performance of the model?[0] https://www.reddit.com/r/LocalLLaMA/comments/140gcn7/new_tok...