hi Can,



 cbalioglu:

When you say “my code freezes” do you mean that your process hangs? Unfortunately I haven’t been able to reproduce this issue. And regarding the torchrun script; you are most likely reading the docs of the master branch. torchrun will be part of v1.10 and is simply an alias to python -m torch.distributed.run.


yes, by simply replacing torch.distributed.launch, which works fine,  with torch.distributed.run the code hangs. in the example i provided above, i removed torch.distributed.barrier that used to test something. i think it is the reason for hanging. because when i type ctrl+c to stop the process, the last line printed in the stack was sleeping. it is like the process was waiting for something. and my guess it is that the process was stuck in the barrier. the same code works fine with launch. i cant run anything right now due to a power outage in our servers. please, give me some time to provide a full example. for torchrun, that explains it why an error was thrown for not recognizing it because i am using torch 1.9.0.



 cbalioglu:

There are some known issues with the launcher scripts in v1.9. I was able to reproduce your problem with stderr not showing up in the log_dir. In the meantime we have fixed most of the issues and a patch release v1.9.1 will be released very soon. Using the latest nightly build, I verified that your script works as expected with proper log output. If this issue is blocking you, I suggest temporarily using a nightly build and migrating to v1.9.1 in a few days once released.


this is good news. thanks. it is not urgent for me right now. but, reading these logs was essential to find the cause of an issue 1 where hints were buried in the first logs that were printed in terminal. i’ll wait for the next release. probably, it could be helpful for others to add this aspect in the doc of 1.9.0. for example, that ddp will turn off multi-threading… unless OMP_NUM_THREADS is explicitly set > 1; this was one of the warnings that i missed because the printing on terminal was fast, and mixed with my own logger.
thanks