dipetkov:

Hello
Have more questions about tokenizers. I enjoyed this section a lot!
This question about WordPiece tokenization. In the compute_pair_scores(splits) function, why add 1 and not freq on line 7?


Thanks for sharing your discovery  and the nice words about the course, indeed I also think it should be freq instead of 1. We will change that in the course.