Just to be clear, for the first option you’ve suggested, I should randomly shuffle the order of each training example when creating the dataset (but drop shuffle=True ), right?

That’s a nice idea too! Just make sure when you continue training that the pre-shuffled dataset is in the same order!

My training data is over a million pairs of sentences extracted from Wikipedia

Woah 0.0
Yea I see the reasoning for this now. Neat problem!