Thanks for the confirmation, @lhoestq! This is very cool.
This style of batched fetching is only used by streaming datasets, right? I’d need to roll my own wrapper to do the same on-the-fly chunking on a local dataset loaded from disk?
And I know it’s not your area, but as far as you know, there’s no way to add/change a dataset’s map functions inside the HF Trainer’s train process, is there? Right now I’ve rigged up resuming from a checkpoint by manually advancing the dataset pre-mapping as I described, reducing the max_steps by the checkpoint number, and setting ignore_date_skip=True in the trainer. It works, but is slightly clunky.