That is weird but it looks like an issue in PyTorch multiprocessing then: setting the num_workers to 0 means they are not creating a new process.
Do you have the issue if you use classic PyTorch DDP or just Accelerate?