Sorry I didn’t realize you are using DDP, so skip the CUDA_LAUNCH_BLOCKING=1 part (it might work, but could also create issues).
Are you sure you are running into a dealock or did a GPU die and the other processes were waiting for the dead process?
If you are using DDP, you could use ps -auxf and check the tree structure for a dead DDP process, when the script is hanging.
If you see that a specific GPU-related process is indeed dead or hanging, you could send a SIGHUP to the main process, so that the logging might continue.
I understand that it’s not easy to create a reproducible code snippet, but if it would be possible, I could take care of the debugging, as it’s not always trivial.