dipetkov:

Comment about the counting subwords in the Unigram tokenization section: (“hug”, 15) should be (“hug”, 5) because “hug” appears as a strict substring only in “hugs”. Unfortunately this affects all the subsequent computations. For example, the sum of all frequencies is 200, not 210.


In this section, what we want to do is create a large initial vocabulary for our example. We choose to put in our initial vocabulary all the strict substrings.
Then, to calculate the frequencies of these tokens in the training dataset we have to take all of them into account, that’s why the number of occurrences of “hug” is 15.
Does this answer your question?