Text Tokenizer
[Text embedding](https://texonom.com/text-embedding-f592251a59af4e5b8a7a70c255546ef7) 


### Text Tokenizer Notion
|Title|
|:-:|
|[Subword Tokenizer](https://texonom.com/subword-tokenizer-9366550159e74e338b914fb2647186fa)|
|[Special Token](https://texonom.com/special-token-cc4caa9ee9d84a1aadf59a545b5d4e1c)|


### Text Tokenizers
|Title|
|:-:|
|[](https://texonom.com/a2a9fb1c6c69477f8c12b6a086e24ded)|
|[tiktoken](https://texonom.com/tiktoken-df0cc79b510742018c269875a4918468)|
|[SubwordTextEncoder](https://texonom.com/subwordtextencoder-2eea9bbc74a94d6cb627b45957b64252)|
|[Huggingface Tokenizer](https://texonom.com/huggingface-tokenizer-15d35355cc9946b1ac0a0e74c1fa1474)|




> [Understanding GPT tokenizers](https://simonwillison.net/2023/Jun/8/gpt-tokenizers/)





