marrrcin:

Can you share a colab link for this? It would be easier to debug.


Sure. Here it is: Google Colab 9



 marrrcin:

Do you use FP16?


No



 marrrcin:

How did you tuned the learning rate?


I just started with 0.001 and tried reducing it. The full attention model was a lot less sensitive to it.



 marrrcin:

Do you use gradient clipping?


Yes, of 1. I’m using the Trainer class from transformers.



 marrrcin:

What is the batch size?



I tried larger values too, and didn’t notice much difference.