Thanks for the great reply! This is very helpful.
Quick questions:


Should I do data preprocessing before launching a distributed training job? In other words, would the ideal case be just having dataset = load_from_disk("path/to/save/directory") in distributed training script?


Is there any built-in prefetch function like the one in e.g. fairseq? Or is this already taken care of?


Thanks for the great reply! This is very helpful.


Quick questions:


Should I do data preprocessing before launching a distributed training job? In other words, would the ideal case be just having dataset = load_from_disk("path/to/save/directory") in distributed training script?


Is there any built-in prefetch function like the one in e.g. fairseq? Or is this already taken care of?


Out of curiosity, how was the shuffling done if not all data is loaded into RAM?