Cade_Gordon:

The following is the log file for the whole run it’s a bit of a behemoth: 32-node-fail.out · GitHub


Thanks for sharing this file, when I looked through that file it seems like there are only 43 workers that report “Added key: store_based_barrier_key:1 to store for rank…”, which lines up with only 43/128 ranks reporting in. It seems like all ranks enter init_process_group but some of them get stuck after that and don’t reach the store based barrier logic to update the store.
Would it be possible to identify one such stuck rank and then attach gdb as follows to see its full stack trace:
gdb -p <pid>
thread apply all bt

Another option is to use the python faulthandler (faulthandler — Dump the Python traceback — Python 3.10.1 documentation 1) to dump the traceback of all threads for a stuck process. dump_traceback_later 1 could be useful here to dump the traceback of a process after X seconds of timeout (indicating that it is stuck).