For token classification, however, there is one potential challenge - alignment. For example, you can’t always tell whether the token is fully correct or partially correct. Additionally, some tokens for a word can have either more corresponding tokens in the correct word or less.
Example : touch and tuch. Suppose these are tokenized into tou,#ch & t, #u, #ch. How will you classify right or wrong in this case?