Thanks for the super helpful response!
Just to be clear, for the first option you’ve suggested, I should randomly shuffle the order of each training example when creating the dataset (but drop shuffle=True), right?
My training data is over a million pairs of sentences extracted from Wikipedia, and if I don’t keep my dataset randomly shuffled, batches will consist of adjacent sentences from the same Wiki document (which I’m trying to avoid). And yes, the reason I’m trying to train my model for only 1 epoch is that I have so much data, which leads to a super long training time with the currently available resource I have.