Correct. Deepspeed or fairscale wasn’t the problem. Can you elaborate a little more on the following?

… more than 1 process compete over your limited resources

So if I understand correctly, if I want to take advantage of fast training using deepspeed and multiple GPUs, is this ‘fighting over limited resources’ inevitable for the preprocessing with map()?