GPT2Tokenizer for Hindi Data
I was trying to fine-tune GPT2LMHeadModel with Hindi data corpus. It is performing well. But when I looked at the tokens that are generated from the GPT2Tokenizer, I saw that they are containing tokens of almost character level. I am not understanding how is this kind of encoding handling Hindi data, or any form of non Roman script correctly. Can anyone explain the working of GPT2Tokenizer from this aspect?
I might be wrong, but I think GPT2Tokenizer uses byte pair encoding, a form of subword-level encoding. On an intuitive level, this is a between character-level and word level, and akin to breaking the word apart by syllable (in reality it's breaking the word apart by the highest frequency patterns). I know some people use Sentence-Piece tokenization for working with Chinese in BERT, so it might be worthwhile to see if there's a similar effort for a Sentence-Piece GPT-2<|||||>@DEBADRIBASAK 
Can you share the steps for fine-tuning on hindi dataset? Thanks.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
