sakasegawa
commited on
Commit
•
bd46c0e
1
Parent(s):
06c5e43
Fix typo
Browse filesNoeX to NeoX
README.md
CHANGED
@@ -82,7 +82,7 @@ for gen_text in tokenizer.batch_decode(gen_tokens, skip_special_tokens=True):
|
|
82 |
The model was trained on [Japanese CC-100](http://data.statmt.org/cc-100/ja.txt.xz), [Japanese Wikipedia](https://dumps.wikimedia.org/other/cirrussearch), and [Japanese OSCAR](https://huggingface.co/datasets/oscar).
|
83 |
|
84 |
# Tokenization
|
85 |
-
The model uses a [special sub-word tokenizer](https://github.com/tanreinama/Japanese-BPEEncoder_V2). Please refer the original repository or [GPT-
|
86 |
|
87 |
# Licenese
|
88 |
[The MIT license](https://opensource.org/licenses/MIT)
|
|
|
82 |
The model was trained on [Japanese CC-100](http://data.statmt.org/cc-100/ja.txt.xz), [Japanese Wikipedia](https://dumps.wikimedia.org/other/cirrussearch), and [Japanese OSCAR](https://huggingface.co/datasets/oscar).
|
83 |
|
84 |
# Tokenization
|
85 |
+
The model uses a [special sub-word tokenizer](https://github.com/tanreinama/Japanese-BPEEncoder_V2). Please refer the original repository or [GPT-NeoX-Japanese](https://huggingface.co/docs/transformers/model_doc/gpt_neox_japanese) in detail.
|
86 |
|
87 |
# Licenese
|
88 |
[The MIT license](https://opensource.org/licenses/MIT)
|