Not sure how you set up your training script, but the common idiom is to use rank 0 for TensorBoard logging; usually after sync’ing workers either implicitly (e.g. start/end of epoch) or explicitly (barrier() call)