Subtokens in BPE in GPT2
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi, I have a question.

For a word represented by more than 1 subtoken in BPE, I wonder what's the principle to divide it into subtokens?

I use GPT2 small version for example. "bookshelf" is divided into "books","he" and "if".(index for encoder is [3835, 258, 1652]).  But actually "book" and "shelf" are represented by single subtoken.(book is 1492, shelf is 18316). Why is "bookshelf" divided into 3 parts instead of "book" and "shelf"?

Thank you!
BPE is a data compression algorithm, the result depends on the statistical properties of the corpus of text on which it has been trained.

You can start with this article https://leimao.github.io/blog/Byte-Pair-Encoding/ to get more of an intuition of how it works.<|||||>> BPE is a data compression algorithm, the result depends on the statistical properties of the corpus of text on which it has been trained.
> 
> You can start with this article https://leimao.github.io/blog/Byte-Pair-Encoding/ to get more of an intuition of how it works.

Thank you! That helps!