Thanks for your replies.
@mrshenli  How can I configure the devices_ids if I want to have one process = one gpu ?
In my case, I have tried to change device_ids by the gpu number (which is equal to the rank value in my case) such as device_ids = [gpu] and call transfer the model in the first gpu model.to(args.gpus[0]) .
But unfortunately, I got that error : RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)
P.S : it seems to “work” now, I forgot to cast the loss function on the good gpu. But, when I increase the number of workers for the dataloader (>0) it creates the error : RuntimeError: DataLoader worker (pid(s) 16451) exited unexpectedly . (and the training took above 24 hours). I will try with the apex distributed and then again compare with DataParallel.