Kamel:

Should I train the tokenizer on the whole corpus which is huge


Yes. With HuggingFace Tokenizers 5, it takes seconds. From the README: “Takes less than 20 seconds to tokenize a GB of text on a server’s CPU”.