Update README.md
Browse files
README.md
CHANGED
@@ -12,26 +12,14 @@ General-purpose Swich transformer based Japanese language model
|
|
12 |
|
13 |
```python
|
14 |
>>> from transformers import AutoModel, AutoTokenizer
|
15 |
-
>>> model = AutoModel.from_pretrained("Tanrei/GPTSAN-japanese")
|
16 |
-
>>> tokenizer = AutoTokenizer.from_pretrained("Tanrei/GPTSAN-japanese")
|
17 |
-
>>> x_tok = tokenizer.encode("武田信玄は、")
|
18 |
-
>>> model = model.cuda()
|
19 |
-
>>> res = model.generator.generate_lm(x_tok, tokenizer)
|
20 |
-
>>> res[0]
|
21 |
-
'勝頼の父であり、天正四年(1576)に死去するまで甲府14万石の大名として甲府を治めた戦国大名ですが...'
|
22 |
-
```
|
23 |
|
24 |
-
## Masked Language Model
|
25 |
-
|
26 |
-
```python
|
27 |
-
>>> from transformers import AutoModel, AutoTokenizer
|
28 |
>>> model = AutoModel.from_pretrained("Tanrei/GPTSAN-japanese")
|
29 |
>>> tokenizer = AutoTokenizer.from_pretrained("Tanrei/GPTSAN-japanese")
|
30 |
-
>>> x_tok = tokenizer.encode("
|
31 |
>>> model = model.cuda()
|
32 |
-
>>>
|
33 |
-
>>>
|
34 |
-
'
|
35 |
```
|
36 |
|
37 |
|
|
|
12 |
|
13 |
```python
|
14 |
>>> from transformers import AutoModel, AutoTokenizer
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
15 |
|
|
|
|
|
|
|
|
|
16 |
>>> model = AutoModel.from_pretrained("Tanrei/GPTSAN-japanese")
|
17 |
>>> tokenizer = AutoTokenizer.from_pretrained("Tanrei/GPTSAN-japanese")
|
18 |
+
>>> x_tok = tokenizer.encode("武田信玄は、", return_tensors="pt")
|
19 |
>>> model = model.cuda()
|
20 |
+
>>> c = model.generate(x_tok.cuda(), max_new_tokens=50, random_seed=63)
|
21 |
+
>>> tokenizer.decode(c[0])
|
22 |
+
'武田信玄は、戦国の頃より「智勇兼備」した英雄として織田信長に比されてきた戦国武将であり、...'
|
23 |
```
|
24 |
|
25 |
|