Loss becoming nearly zero in first 5K steps when training LM from scratch
I am training the ALBERT LM model from scratch.

I have already trained it for Hindi and Bangla and it was working fine but when I am training on Gujarati, the loss is becoming zero in 5K steps.

What could be the reason for the sudden drop in the loss? Can anyone suggest what could be the cause or how to debug such issue? 
Any suggestions?


Hi! This is an interesting question, have you tried asking it over on the forums at https://discuss.huggingface.co ? You'll probably get more answers over there.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
