Recently, I have been using bert-base-multilingual-cased for my work in bengali. When I feed in a sentence like “আজকে হবে না” to BertTokenizer, I get the following output.
Sentence:  আজকে হবে না
Tokens:  ['আ', '##জ', '##কে', 'হবে', 'না']
To Int:  [938, 24383, 18243, 53761, 26109]

But when I fed in a sentence like “আজকে হবেনা” with “না” not spaced, I see tokens been “##না” with corresponding index also been changed
Sentence:  আজকে হবেনা
Tokens:  ['আ', '##জ', '##কে', 'হবে', '##না']
To Int:  [938, 24383, 18243, 53761, 20979]

Now, I was hoping is there anyway to let tokenizer to know that if they find anything like ‘##না’, convert them to ‘না’ for all such cases.