@lewtun Thank you for your response. The loss does decrease but the model does not learn anything useful. The task is a sequence to sequence task, where the encoder and decoder are both roberta models. For the “bad” models the generated output is usually something like:
the="="="="="="="="="="="="="="="="="="="="="="="="="="="="="="="=" The the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the.... (Despite the fact that we set the no_repeat_ngram_size =3 ). The point about the warmup step is valid, and maybe we should increase it. I will try with that. What is strange, is that the training set up for the smaller sample (500K) and the larger (2M) are exactly the same, the only difference is the data. Therefore such instability is very strange.