Thx for reply, no background process was found and the port was always available.
I have fixed a typo in my command, where --master_addr ='10.0.3.29' --master_port=9901 had been typed as --master_addr ='10.0.3.29' --master_port='10.0.3.29'.
However the error has remained and at least one process has launched with the output
Traceback (most recent call last):
  File "mnist-dist.py", line 203, in <module>
    init_processes(args.rank, args.world_size, run, args.batch_size, backend=args.backend)
  File "mnist-dist.py", line 198, in init_processes
    dist.init_process_group(backend=backend, world_size=world_size, rank=rank, init_method="env://")
  File "/home/dl/anaconda2/envs/torch-dist-py3.6/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 354, in init_process_group
    store, rank, world_size = next(rendezvous(url))
  File "/home/dl/anaconda2/envs/torch-dist-py3.6/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 143, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, start_daemon)
RuntimeError: Address already in use
dl-Server:10648:10648 [1] NCCL INFO NET : Using interface enp0s31f6:10.0.13.29<0>
dl-Server:10648:10648 [1] NCCL INFO NET/IB : Using interface enp0s31f6 for sideband communication
dl-Server:10648:10648 [1] NCCL INFO Using internal Network Socket
dl-Server:10648:10648 [1] NCCL INFO NET : Using interface enp0s31f6:10.0.13.29<0>
dl-Server:10648:10648 [1] NCCL INFO NET/Socket : 1 interfaces found
NCCL version 2.3.7+cuda9.0
dl-Server:10648:10648 [1] NCCL INFO rank 0 nranks 1
dl-Server:10648:10675 [1] NCCL INFO comm 0x7fd38c00d3f0 rank 0 nranks 1
dl-Server:10648:10675 [1] NCCL INFO CUDA Dev 1, IP Interfaces : enp0s31f6(PHB) 
dl-Server:10648:10675 [1] NCCL INFO Using 256 threads
dl-Server:10648:10675 [1] NCCL INFO Min Comp Cap 6
dl-Server:10648:10675 [1] NCCL INFO comm 0x7fd38c00d3f0 rank 0 nranks 1 - COMPLETE
Train Epoch 1 - 0 / 234         Loss  5.576234
Train Epoch 1 - 10 / 234        Loss  5.541703
Train Epoch 1 - 20 / 234        Loss  5.515630
Train Epoch 1 - 30 / 234        Loss  5.514045
Train Epoch 1 - 40 / 234        Loss  5.485974
Train Epoch 1 - 50 / 234        Loss  5.462833
Train Epoch 1 - 60 / 234        Loss  5.422739
Train Epoch 1 - 70 / 234        Loss  5.374931
Train Epoch 1 - 80 / 234        Loss  5.342307
Train Epoch 1 - 90 / 234        Loss  5.291063
Train Epoch 1 - 100 / 234       Loss  5.220443
Train Epoch 1 - 110 / 234       Loss  5.083968
Train Epoch 1 - 120 / 234       Loss  5.002171
Train Epoch 1 - 130 / 234       Loss  4.953607
Train Epoch 1 - 140 / 234       Loss  4.894170
Train Epoch 1 - 150 / 234       Loss  4.805832
Train Epoch 1 - 160 / 234       Loss  4.792961
Train Epoch 1 - 170 / 234       Loss  4.732522
Train Epoch 1 - 180 / 234       Loss  4.770869
Train Epoch 1 - 190 / 234       Loss  4.688779
Train Epoch 1 - 200 / 234       Loss  4.725927
Train Epoch 1 - 210 / 234       Loss  4.620460
Train Epoch 1 - 220 / 234       Loss  4.605740
Train Epoch 1 - 230 / 234       Loss  4.563363
Epoch 1 of took 5.263s
Current Total time : 5.263s
Train Epoch 2 - 0 / 234         Loss  4.555817
Train Epoch 2 - 10 / 234        Loss  4.603082
Train Epoch 2 - 20 / 234        Loss  4.618500
Train Epoch 2 - 30 / 234        Loss  4.520389
Train Epoch 2 - 40 / 234        Loss  4.531864
Train Epoch 2 - 50 / 234        Loss  4.467782
Train Epoch 2 - 60 / 234        Loss  4.447100
Train Epoch 2 - 70 / 234        Loss  4.424728
Train Epoch 2 - 80 / 234        Loss  4.433639
Train Epoch 2 - 90 / 234        Loss  4.372109
Train Epoch 2 - 100 / 234       Loss  4.435561
Train Epoch 2 - 110 / 234       Loss  4.351253
Train Epoch 2 - 120 / 234       Loss  4.306677
Train Epoch 2 - 130 / 234       Loss  4.343150
Train Epoch 2 - 140 / 234       Loss  4.243150
Train Epoch 2 - 150 / 234       Loss  4.347620
Train Epoch 2 - 160 / 234       Loss  4.217095
Train Epoch 2 - 170 / 234       Loss  4.255800
Train Epoch 2 - 180 / 234       Loss  4.282191
Train Epoch 2 - 190 / 234       Loss  4.249407
Train Epoch 2 - 200 / 234       Loss  4.209113
Train Epoch 2 - 210 / 234       Loss  4.194527
Train Epoch 2 - 220 / 234       Loss  4.220213
Train Epoch 2 - 230 / 234       Loss  4.201759
Epoch 2 of took 5.524s
Current Total time : 10.787s
Train Epoch 3 - 0 / 234         Loss  4.158279
Train Epoch 3 - 10 / 234        Loss  4.111032
Train Epoch 3 - 20 / 234        Loss  4.147989
Train Epoch 3 - 30 / 234        Loss  4.255434
Train Epoch 3 - 40 / 234        Loss  4.111946
Train Epoch 3 - 50 / 234        Loss  4.111733
Train Epoch 3 - 60 / 234        Loss  4.176547
Train Epoch 3 - 70 / 234        Loss  4.063233
Train Epoch 3 - 80 / 234        Loss  4.079793
Train Epoch 3 - 90 / 234        Loss  4.042555
Train Epoch 3 - 100 / 234       Loss  4.050662
Train Epoch 3 - 110 / 234       Loss  4.066662
Train Epoch 3 - 120 / 234       Loss  4.090621
Train Epoch 3 - 130 / 234       Loss  4.015823
Train Epoch 3 - 140 / 234       Loss  4.092526
Train Epoch 3 - 150 / 234       Loss  4.045942
Train Epoch 3 - 160 / 234       Loss  4.048071
Train Epoch 3 - 170 / 234       Loss  3.984233
Train Epoch 3 - 180 / 234       Loss  3.942847
Train Epoch 3 - 190 / 234       Loss  3.943717
Train Epoch 3 - 200 / 234       Loss  3.959996
Train Epoch 3 - 210 / 234       Loss  4.059554
Train Epoch 3 - 220 / 234       Loss  3.918130
Train Epoch 3 - 230 / 234       Loss  4.074725
Epoch 3 of took 5.308s
Current Total time : 16.095s
Train Epoch 4 - 0 / 234         Loss  3.944645
Train Epoch 4 - 10 / 234        Loss  3.923414
Train Epoch 4 - 20 / 234        Loss  3.944232
Train Epoch 4 - 30 / 234        Loss  3.978234
Train Epoch 4 - 40 / 234        Loss  3.950741
Train Epoch 4 - 50 / 234        Loss  3.913695
Train Epoch 4 - 60 / 234        Loss  3.907088
Train Epoch 4 - 70 / 234        Loss  4.026055
Train Epoch 4 - 80 / 234        Loss  3.854659
Train Epoch 4 - 90 / 234        Loss  3.954557
Train Epoch 4 - 100 / 234       Loss  3.880200
Train Epoch 4 - 110 / 234       Loss  3.911777
Train Epoch 4 - 120 / 234       Loss  3.866536
Train Epoch 4 - 130 / 234       Loss  3.957554
Train Epoch 4 - 140 / 234       Loss  3.930515
Train Epoch 4 - 150 / 234       Loss  3.950871
Train Epoch 4 - 160 / 234       Loss  3.845739
Train Epoch 4 - 170 / 234       Loss  3.905876
Train Epoch 4 - 180 / 234       Loss  3.884211
Train Epoch 4 - 190 / 234       Loss  4.034623
Train Epoch 4 - 200 / 234       Loss  3.863284
Train Epoch 4 - 210 / 234       Loss  3.899471
Train Epoch 4 - 220 / 234       Loss  3.837218
Train Epoch 4 - 230 / 234       Loss  3.862398
Epoch 4 of took 5.271s
Current Total time : 21.366s
Train Epoch 5 - 0 / 234         Loss  3.878444
Train Epoch 5 - 10 / 234        Loss  3.919256
Train Epoch 5 - 20 / 234        Loss  3.872842
Train Epoch 5 - 30 / 234        Loss  3.926296
Train Epoch 5 - 40 / 234        Loss  3.787506
Train Epoch 5 - 50 / 234        Loss  3.959824
Train Epoch 5 - 60 / 234        Loss  3.830777
Train Epoch 5 - 70 / 234        Loss  3.883856
Train Epoch 5 - 80 / 234        Loss  3.877614
Train Epoch 5 - 90 / 234        Loss  3.846863
Train Epoch 5 - 100 / 234       Loss  3.908530
Train Epoch 5 - 110 / 234       Loss  3.819784
Train Epoch 5 - 120 / 234       Loss  3.798816
Train Epoch 5 - 130 / 234       Loss  3.757388
Train Epoch 5 - 140 / 234       Loss  3.837136
Train Epoch 5 - 150 / 234       Loss  3.855000
Train Epoch 5 - 160 / 234       Loss  3.821057
Train Epoch 5 - 170 / 234       Loss  3.777124
Train Epoch 5 - 180 / 234       Loss  3.714392
Train Epoch 5 - 190 / 234       Loss  3.776406
Train Epoch 5 - 200 / 234       Loss  3.886733
Train Epoch 5 - 210 / 234       Loss  3.927509
Train Epoch 5 - 220 / 234       Loss  3.719052
Train Epoch 5 - 230 / 234       Loss  3.785564
Epoch 5 of took 5.216s
Current Total time : 26.582s

Test set : Average loss: 4.8523, Accuracy: 9453/10000 (94%)

Average epoch time(ex. 1.) : 5.330s
Total time : 26.582s


Would you like to have a look ?