ph0123:

I submit the batch file. In the batch file, I can get the node names of all nodes, which assigned by the server.
For each submission, the node names will be changed.


In that case, I wonder if it would be possible to programmably figure out the master. E.g., ask all processes to sort all node names and then always use the first one?

How to set up the time out for this situation. The program will run with 20-30 minutes.

Are you using RPC or DDP? For RPC, checkout this PR (https://github.com/pytorch/pytorch/pull/38577 2). For DDP, init_process_group does take a timeout argument.