fine-tuning BERT, next sentence prediction loss is not decreasing
I run simple_lm_finetuning and monitor the loss change of next_sentence_loss and masked_lm_loss. The loss of masked token can converge but the next_sentence_loss is not decreasing.

For now I tried: tuning learning rate, change to optimizers from pytorch.optim, I checked the input_ids and the input looks good... I also ignore the loss from masked tokens and specifically train next sentence prediction but it didn't work.

 Did anyone face the same problem? Or where should I check to make it work? Thanks.




This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>same not working for me. NSP loss is not converging even though MLM loss is converging.