Instead of pre-processing, you can also do this:

Mapping in a distributed setting

In a distributed setting, you may use caching and a torch.distributed.barrier() to make sure that only the main process performs the mapping, while the other ones load its results. This avoids duplicating work between all the processes, or worse, requesting more CPUs than your system can handle.
continued at:
  
      

      huggingface.co
  

  
    

Processing data in a Dataset 11

  🤗Datasets provides many methods to modify a Dataset, be it to reorder, split or shuffle the dataset or to apply data processing functions or evaluation funct...