Understanding GPT tokenizers
For those who interested, there some new researches in the field [0]. It usually possible to create more compact token representation from given text, but my guess the greedy "optimal" tokenizer might harm the performance of the model?<p>[0] <a href="https://www.reddit.com/r/LocalLLaMA/comments/140gcn7/new_tokenizer_increases_inference_speed_and/" rel="nofollow noreferrer">https://www.reddit.com/r/LocalLLaMA/comments/140gcn7/new_tok...</a>