Hi all,
I have questions on albert-base-v1 1 and v2 models uploaded in the huggingface model hub. I’ve checked the MLM loss of albert base models using book corpus dataset and squad context data(which is basically similar to Wikipedia data) based on this example script from transformer repo, in order to validate initial model performance (without additional training). It appears that the average MLM losses are around 2.5(squad context) and 3.2(book corpus) in albert-base-v1, which is much worse than I expected given that those two datasets must have been used for the pertaining albert base model. The value was much worse in v2.
I would like to ask whether albert-base models are trained until the losses converge, or the model has been trained on one or two epochs. Also, it would be great if there is an actual training script and/or MLM loss history on albert pertaining models.
Thanks!
Hojin