Hey @pjahn89, I am facing a similar issue while trying to finetuning T5 on XSum using TPU/GPU. The training loss is not constant (it varies, but doesn’t converge). But, my validation loss is constant, like literally not even a change in 5th decimal place, I tried many things like creating my nn.Module compatible with the trainer. Subclassed the trainer to modify compute_loss(). But, I am not seeing any change. If you have solved this issue, can you please tell me how?
Here’s a link 5 to my colab for reference.
Thank you!