comet-gpt2-xl-japanese / special_tokens_map.json
Eiki's picture
Upload tokenizer
7252eed
raw
history blame
189 Bytes
{
"additional_special_tokens": [
"xNeed",
"xEffect",
"xIntent",
"xReact"
],
"bos_token": "<s>",
"eos_token": "</s>",
"pad_token": "</s>",
"unk_token": "<unk>"
}