Two things I’ve realized already:


The tokenizing step should pad each example to the max_length, since examples from different dataset batches could end up in the same training batch.


Resuming training spends a long time iterating up to the first batch, since the mapped function is called even if the data is just getting skipped, and it’s a non-trivial amount of work. It’s much faster to manually skip batches before the map is applied. But it’s hard to know how far ahead to skip, since there’s no record of which training examples came from which dataset examples. So I split my mapped function in two. The first splits the long data into manageable chunks. I apply that map, manually skip to the checkpoint, and then apply a second map, which has the preprocessing/tokenizing/noising code.


But other than those two things, my initial tests with this seem to be working really well so far.