readme: ---
  
  language: ja
  
  license: cc-by-sa-4.0
  
  tags:
  
  - finance
  
  widget:
  
  - text: 流動[MASK]は、1億円となりました。
  
  ---
  
  # Additional pretrained BERT base Japanese finance 
  
  This is a [BERT](https://github.com/google-research/bert) model pretrained on texts in the Japanese language.
  
  The codes for the pretraining are available at [retarfi/language-pretraining](https://github.com/retarfi/language-pretraining/tree/v1.0).
  
  ## Model architecture
  
  The model architecture is the same as BERT small in the [original BERT paper](https://arxiv.org/abs/1810.04805); 12 layers, 768 dimensions of hidden states, and 12 attention heads.
  
  ## Training Data
  
  The models are additionally trained on financial corpus from [Tohoku University's BERT base Japanese model (cl-tohoku/bert-base-japanese)](https://huggingface.co/cl-tohoku/bert-base-japanese).
  
  The financial corpus consists of 2 corpora:
  
  - Summaries of financial results from October 9, 2012, to December 31, 2020
  - Securities reports from February 8, 2018, to December 31, 2020
  
  The financial corpus file consists of approximately 27M sentences.
  
  
  ## Tokenization
  
  You can use tokenizer [Tohoku University's BERT base Japanese model (cl-tohoku/bert-base-japanese)](https://huggingface.co/cl-tohoku/bert-base-japanese).
  
  You can use the tokenizer:
  
  ```
  tokenizer = transformers.BertJapaneseTokenizer.from_pretrained('cl-tohoku/bert-base-japanese')
  ```
  
  ## Training
  
  The models are trained with the same configuration as BERT base in the [original BERT paper](https://arxiv.org/abs/1810.04805); 512 tokens per instance, 256 instances per batch, and 1M training steps.
  
  ## Citation
  
  ```
  @article{Suzuki-etal-2023-ipm,
    title = {Constructing and analyzing domain-specific language model for financial text mining}
    author = {Masahiro Suzuki and Hiroki Sakaji and Masanori Hirano and Kiyoshi Izumi},
    journal = {Information Processing & Management},
    volume = {60},
    number = {2},
    pages = {103194},
    year = {2023},
    doi = {10.1016/j.ipm.2022.103194}
  }
  ```
  
  ## Licenses
  
  The pretrained models are distributed under the terms of the [Creative Commons Attribution-ShareAlike 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
  
  ## Acknowledgments
  
  This work was supported by JSPS KAKENHI Grant Number JP21K12010 and JST-Mirai Program Grant Number JPMJMI20B1.
  