readme: ---
  language:
  - en
  - ja
  license: mit
  datasets:
  - snow_simplified_japanese_corpus
  tags:
  - ja
  - japanese
  - tokenizer
  widget:
  - text: "誰が一番に着くか私には分かりません。"
  ---
  
  # Japanese Dummy Tokenizer
  
  Repository containing a dummy Japanese Tokenizer trained on ```snow_simplified_japanese_corpus``` dataset. The tokenizer has been trained using Hugging Face datasets in a streaming manner. 
  
  ## Intended uses & limitations
  
  You can use this tokenizer to tokenize Japanese sentences.
  
  ## How to use it
  
  ```
  from transformers import AutoTokenizer
  
  tokenizer = AutoTokenizer.from_pretrained("ybelkada/japanese-dummy-tokenizer")
  ```
  
  ## How to train the tokenizer
  
  Check the file ```tokenizer.py```, you can freely adapt it to other datasets. This tokenizer is based on the tokenizer from ```csebuetnlp/mT5_multilingual_XLSum```.