readme: ---
  language: ja
  tags:
  - t5
  - text2text-generation
  - seq2seq
  license: apache-2.0
  datasets:
  - mc4
  - wiki40b
  ---
  
  # t5-base-japanese-web (with Byte-fallback, 8K)
  
  ## Description
  
  [megagonlabs/t5-base-japanese-web](https://huggingface.co/megagonlabs/t5-base-japanese-web) is a T5 (Text-to-Text Transfer Transformer) model pre-trained on Japanese web texts.  
  Training codes are [available on GitHub](https://github.com/megagonlabs/t5-japanese).
  
  The vocabulary size of this model is 8K.
  [32K version is also available](https://huggingface.co/megagonlabs/t5-base-japanese-web).
  
  ### Corpora
  
  We used following corpora for pre-training.
  
  - Japanese in [mC4/3.0.1](https://huggingface.co/datasets/mc4) (We used [Tensorflow native format](https://github.com/allenai/allennlp/discussions/5056))
      - 87,425,304 pages
      - 782 GB in TFRecord format
  - [Japanese](https://www.tensorflow.org/datasets/catalog/wiki40b#wiki40bja) in [wiki40b/1.3.0](https://www.tensorflow.org/datasets/catalog/wiki40b)
      - 828,236 articles (2,073,584 examples)
      - 2 GB in TFRecord format
  
  ### Tokenizer
  
  We used Japanese Wikipedia to train [SentencePiece](https://github.com/google/sentencepiece).
  
  - Vocabulary size: 8,000
  - [Byte-fallback](https://github.com/google/sentencepiece/releases/tag/v0.1.9): Enabled
  
  ### Parameters
  
  - T5 model: [models/t5.1.1.base.gin](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/t5/models/gin/models/t5.1.1.base.gin)
  - Training steps: 1,000,000
  
  It took about 126 hours with TPU v3-8
  
  ## Related models
  
  - [日本語T5事前学習済みモデル (sonoisa/t5-base-japanese)](https://huggingface.co/sonoisa/t5-base-japanese)
  - [日本語T5事前学習済みモデル (sonoisa/t5-base-japanese-mC4-Wikipedia)](https://huggingface.co/sonoisa/t5-base-japanese-mC4-Wikipedia)
  
  ## License
  
  Apache License 2.0
  
  ## Citations
  
  - mC4
  
  Contains information from `mC4` which is made available under the [ODC Attribution License](https://opendatacommons.org/licenses/by/1-0/).
  
  ```bibtex
  @article{2019t5,
      author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
      title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
      journal = {arXiv e-prints},
      year = {2019},
      archivePrefix = {arXiv},
      eprint = {1910.10683},
  }
  ```
  
  - wiki40b
  
  ```bibtex
  @inproceedings{49029,
  title = {Wiki-40B: Multilingual Language Model Dataset},
  author = {Mandy Guo and Zihang Dai and Denny Vrandecic and Rami Al-Rfou},
  year = {2020},
  booktitle   = {LREC 2020}
  }
  ```
  