Bidirectional Encoder Representations from Transformers (BERT) models for biomedical specialties such as BioBERT and clinicalBERT  have significantly improved in biomedical text-mining tasks and enabled us to extract valuable information from biomedical literature . However, we benefitted only in English because of the significant scarcity of high-quality medical documents, such as PubMed, in each language. Therefore, we propose a method that realizes a high-performance BERT model by using a small corpus. We introduce the method to train a BERT model on a small medical corpus both in English and Japanese, respectively, and then we evaluate each of them in terms of the biomedical language understanding evaluation (BLUE) benchmark and the medical-document-classification task in Japanese, respectively. After confirming their satisfactory performances, we apply our method to develop a model that outperforms the pre-existing models. Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University (ouBioBERT) achieves the best scores on 7 of the 10 datasets in terms of the BLUE benchmark. The total score is 1.0 points above that of BioBERT  .
Bidirectional Encoder Representations from Transformers (BERT) models for medical specialties, such as BioBERT and clinicalBERT , have significantly improved in performing biomedical text mining tasks and have enabled extracting valuable information from biomedical literature ; however, only English speakers benefit due to the significant scarcity of high-quality medical documents, such as PubMed, in each language. Therefore, we propose a method to train a high-performance BERT model using a small corpus. We introduce the method to train a BERT model on a small medical corpus both in English and in Japanese, and we present the evaluation of each model in terms of the biomedical language understanding evaluation (BLUE) benchmark and the medical document classification task in Japanese, respectively. After confirming their satisfactory performances, we applied our method to develop a model comparable to the publicly available models. OuBioBERT, short for Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University , achieved the best score in terms of the BLUE benchmark. The total score is 1.1 points above that of BioBERT and 0.3 points above that of the ablated model trained without our proposed method. This proposed technique is an effective approach to develop localized medical BERT models and to enhance domain-specific models in the biomedical domain .
R
biomedical specialties
medical specialties,
A
,
R
biomedical text-mining tasks and enabled us to extract
performing biomedical text mining tasks and have enabled extracting
R
. However, we benefitted only in English because of
; however, only English speakers benefit due to
R
that realizes
to train
D
by
R
Japanese, respectively, and then we evaluate each of them
in Japanese, and we present the evaluation of each model
R
medical-document-classification
medical document classification
R
apply
applied
R
that outperforms the pre-existing models.
comparable to the publicly available models. OuBioBERT, short for
R
(ouBioBERT) achieves the best scores on 7 of the 10 datasets
, achieved the best score
R
1.0
1.1
A
and 0.3 points above that of the ablated model trained without our proposed method. This proposed technique is an effective approach to develop localized medical BERT models and to enhance domain-specific models in the biomedical domain