Thank you for your quick reply. It is reproducible. Refer to https://www.kaggle.com/mhviraf/qa-bert-pytorch?scriptVersionId=28476451; 2 versions 2 and 8 work perfectly fine. versions 3 through 7 doesn’t. I don’t know how familiar you are with Kaggle notebooks but if you click on the “8 commits” link on the top left of the page it lets you see the difference between versions 7 and 8.
I ran the original version twice and both times the mean_spearman goes up to 0.54 after about 10 epochs. Whilst, I ran the modified version where the loss is returned for 5 times and every time spear_mean jumps back and forth between -0.05 and 0.16 through epochs.
One thing that is very suspicious is that when I return the loss it iterates very fast over epochs and the runtime becomes about 10sec/epoch while in the original code the runtime is about 400sec/epoch.