nlp-waseda

/

roberta-base-japanese

like 32

Dear model creators,
It seems like this model doesn't include LM layers:

>>> model = AutoModelForMaskedLM.from_pretrained("nlp-waseda/roberta-base-japanese")
Some weights of RobertaForMaskedLM were not initialized from the model checkpoint at nlp-waseda/roberta-base-japanese and are newly initialized: ['lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.bias']

which makes impossible to do further fine-tuning on domain-specific data.
If possible, could you please include these layers to the base model. Large model doesn't have this problem, but infeasible to use due to GPU memory constraints.