Thank you for you response.
I already tested with less number of workers, and to be sure I tested once again after your post. It takes more time with less workers (I set it to 10). My machine has 20 physical cores, it is NVIDIA DGX Station.
I tried:

torch.distributed.init_process_group(backend=“nccl”)
model = torch.nn.parallel.DistributedDataParallel(model)

but I got the following error:

ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable RANK expected, but not set

If I understand it well, DataParallel 3 does the same stuff or it is wrong?