Unfortunately it doesn’t seem to be as simple as I thought. Also, it doesn’t seem consisten? Doing some tests:

cli flag 512, config.json 514: I’m getting the error again.
cli 512, config 1024: crashes with the same error
cli 256: config 514: the training loop runs.
cli 207, config 514: the training loop runs.
cli 500, config 514: training loop runs
cli 508, config 514: training loop runs
cli 512, config 514 again: same error again.
cli 508, config 514: runs fine again

Tried rm -rf ~/.cache/huggingface/datasets and ran some more tests:

512, 514: I see some progress bars, as it apparently reruns some dataset stuff… then I get the indexError again.
512, 516: error
512, 700: error
512, 1024: error
512, 1028: error
510, 514: two progress bars again, as it does some sort of dataset processing, then training loop runs fine.
513, 514: progress bars and then error.
511, 514: progress bars and error
510, 514: trains fine
510, 1024: trains fine
510, 256: trains fine!!

I’m stumped. @sgugger got any ideas?