Ah yes, I see it in the paper now. Never really thought about that. It seems that they did this sequentially, so first training on 128 seq_length on 90% of the data, and only then train further for 512. So I would probably do it like that, too. First train the 128-part, and then continue training with a new trainer.