|
--- |
|
language: ja |
|
license: cc-by-sa-4.0 |
|
datasets: |
|
- wikipedia |
|
- cc100 |
|
- oscar |
|
mask_token: "[MASK]" |
|
widget: |
|
- text: "[MASK] 大学 で 自然 言語 処理 を 学ぶ 。" |
|
--- |
|
|
|
# nlp-waseda/bigbird-base-japanese |
|
|
|
## Model description |
|
|
|
This is a Japanese BigBird base model pretrained on Japanese Wikipedia, the Japanese portion of CC-100, and the Japanese portion of OSCAR. |
|
|
|
## How to use |
|
|
|
You can use this model for masked language modeling as follows: |
|
```python |
|
from transformers import AutoTokenizer, AutoModelForMaskedLM |
|
tokenizer = AutoTokenizer.from_pretrained("nlp-waseda/bigbird-base-japanese") |
|
model = AutoModelForMaskedLM.from_pretrained("nlp-waseda/bigbird-base-japanese") |
|
|
|
sentence = '[MASK] 大学 で 自然 言語 処理 を 学ぶ 。' # input should be segmented into words by Juman++ in advance |
|
encoding = tokenizer(sentence, return_tensors='pt') |
|
... |
|
``` |
|
|
|
You can fine-tune this model on downstream tasks. |
|
|
|
## Tokenization |
|
|
|
The input text should be segmented into words by [Juman++](https://github.com/ku-nlp/jumanpp) in advance. Juman++ 2.0.0-rc3 was used for pretraining. Each word is tokenized into tokens by [sentencepiece](https://github.com/google/sentencepiece). |
|
|
|
## Vocabulary |
|
|
|
The vocabulary consists of 32000 tokens including words ([JumanDIC](https://github.com/ku-nlp/JumanDIC)) and subwords induced by the unigram language model of [sentencepiece](https://github.com/google/sentencepiece). |
|
|
|
## Training procedure |
|
|
|
This model was trained on Japanese Wikipedia (as of 20221101), the Japanese portion of CC-100, and the and the Japanese portion of OSCAR. It took two weeks using 16 NVIDIA A100 GPUs using [transformers](https://github.com/huggingface/transformers) and [DeepSpeed](https://github.com/microsoft/DeepSpeed). |
|
|
|
The following hyperparameters were used during pretraining: |
|
- learning_rate: 1e-4 |
|
- per_device_train_batch_size: 6 |
|
- gradient_accumulation_steps: 2 |
|
- total_train_batch_size: 192 |
|
- max_seq_length: 4096 |
|
- training_steps: 600000 |
|
- warmup_steps: 6000 |
|
- bf16: true |
|
- deepspeed: [ds_config.json](https://huggingface.co/nlp-waseda/bigbird-base-japanese/blob/main/ds_config.json) |
|
|
|
## Performance on JGLUE |
|
|
|
We fine-tuned the following models and evaluated them on the dev set of JGLUE. |
|
We tuned learning rate and training epochs for each model and task following [the JGLUE paper](https://www.jstage.jst.go.jp/article/jnlp/30/1/30_63/_pdf/-char/ja). |
|
|
|
For the tasks other than MARC-ja, the maximum length is short, so the attention_type was set to "original_full", and fine-tuning was performed. For MARC-ja, both "block_sparse" and "original_full" were used. |
|
|
|
| Model | MARC-ja/acc | JSTS/pearson | JSTS/spearman | JNLI/acc | JSQuAD/EM | JSQuAD/F1 | JComQA/acc | |
|
|-------------------------------|--------------|---------------|----------|-----------|-----------|------------|------------| |
|
| Waseda RoBERTa base | 0.965 | 0.913 | 0.876 | 0.905 | 0.853 | 0.916 | 0.853 | |
|
| Waseda RoBERTa large (seq512) | 0.969 | 0.925 | 0.890 | 0.928 | 0.910 | 0.955 | 0.900 | |
|
| BigBird base (original_full) |0.959 | 0.888 | 0.846 | 0.896 | 0.884 | 0.933 | 0.787 | |
|
| BigBird base (block_sparse) |0.959 | - | - | - | - | - | - | |
|
|
|
## Acknowledgments |
|
|
|
This work was supported by AI Bridging Cloud Infrastructure (ABCI) through the "Construction of a Japanese Large-Scale General-Purpose Language Model that Handles Long Sequences" at the 3rd ABCI Grand Challenge 2022. |
|
|
