Optimize tokenization (85-92% time reduction)
This optimization in tokenization reduces the tokenization time by ~ 85-92 %

Stats:

Before fix: 
 8%|████████▉                                                                                                      | 119574/1486030 [01:03<12:09, 1874.22it/s]

After fix: 
 98%|████████████████████████████████████  | 1459590/1486030 [01:59<00:02, 12244.82it/s]


Another stats: 
When huge number of new tokens (~58K) are added, tokenization time bumps up to ~27-28 hours 
With this fix the time drops to ~2.15 hours !
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
