I apply the tokenizer to my custom dataset using the  datasets.Dataset.map()  4 method as done in the run_mlm.py 3  example. FYI, I am using multiprocessing by setting num_proc parameter of map().
I have a multi-GPU system, and doing the above usually takes about ~10 minutes. But once I use DeepSpeed (deepspeed --include localhost:0,1,2), the process takes much longer (~20 minutes). I’m aware the preprocessing is done 100% on the CPU, so I don’t know why I am experiencing this slow down in speed in the first place. If I limit the number of visible devices to just one (deepspeed --include localhost:0), the preprocessing using map() seems to be fast again.