Thanks @BramVanroy! That is what I thought. The weights will not be relevant. How do you suggest I solve this?  I was thinking of trying  to train both the sequence classification task and the mlm task on my dataset in a multi-task setting. Not sure it you or anyone has any pointers