f10w:

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.


This is printed immediately after you run launch.py. See the code below:

  
      github.com
  
  
    pytorch/pytorch/blob/0fc0a9308ab1e8f6b65d2565c45f54afb449d438/torch/distributed/launch.py#L215-L222 19

if 'OMP_NUM_THREADS' not in os.environ and args.nproc_per_node > 1:
    current_env["OMP_NUM_THREADS"] = str(1)
    print("*****************************************\n"
          "Setting OMP_NUM_THREADS environment variable for each process "
          "to be {} in default, to avoid your system being overloaded, "
          "please further tune the variable for optimal performance in "
          "your application as needed. \n"
          "*****************************************".format(current_env["OMP_NUM_THREADS"]))



  
  
    
    
  
  


During the few hours when the job is running, did the DistributedDataParallel training making progress as expected? You can, e.g., print some logs in every iteration to check this.
And since the command line contains a path to checkpoint, I assume the training job would write into that checkpoint file periodically? Did the job successfully generated any checkpoint?