I am using Distributed Data Parallel wich instantiates multiple processes to train model on multiple GPUs. I want to be able to save each experiment run backup to a single new folder (let’s say by passing the same timestamp to all the processes). However some processes being delayed by a second leads to a different timestamp within the same experiment run. Is it possible to pass the same information(single timestamp) to all the processes?
I start my script with:
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 NCCL_LL_THRESHOLD=0 python \
-i \
-m torch.distributed.launch \
--master_port=9997 \
--nproc_per_node=8 \
main.py .....

Thanks