Greetings
Would like to contribute to Amharic (less-resource language) I noticed resources for this language is next to nil.
Where do I start and build up?
1st datasets?
2nd tokenizers ?
3rd transformers?

  
      github.com/huggingface/tokenizers
  
  
    
  
	  
  

  
    
      ByteLevelBPETokenizer output seems weird 8
    

    
      
        opened 
        
          
        
        Mar 24, 2020
      
      


      
        
          
          seyyaw
        
      
    
  



  I use the ByteLevelBPETokenizer to train a custom tokenizer for Amharic language (less-resource language).
tokenizer = ByteLevelBPETokenizer(lowercase=False)
tokenizer.train(files=paths, vocab_size=32000, min_frequency=3, special_tokens=[
 "<s>",
 "<pad>",
...





  
  
    
    
  
  


Thanks!