Hello all,
I have just started exploring the tokenizer library and I want to know what know what are the basic normalizer steps which are performed in BPE tokenizer . Is there any way I can list out the contents of
tokenizer.normarlizer
Regards
Sumeet