Hi @ptrblck,
This issue still exists though it seems fixes were pushed in the issues you cited.
I am using Pytorch 1.9.0 CUDA 10.2 build and Python 3.7 on Linux.
Do you know how and where to initialize the logger so that it works correctly?
Thanks.
EDIT: Seems like I found an implementation which works. I have been scouring for a solution where logging works with torch.distributed and couldn’t seem to make it work. I also saw some other users facing issues with the same thing where the logger wouldn’t print anything to console/file.
I looked at the logger implementation here
  

      github.com
  

  
    stanford-futuredata/pytorch-distributed/blob/c8c25e82b4f49ff8a4139283d1d47a435ede04c6/training/logger.py 9


    from tensorboardX import SummaryWriter
import torch
import time    
  
class TensorboardLogger:
  def __init__(self, output_dir, is_master=False):
    self.output_dir = output_dir
    self.current_step = 0
    if is_master: self.writer = SummaryWriter(self.output_dir)
    else: self.writer = NoOp()
    self.log('first', time.time())

  def log(self, tag, val):
    """Log value to tensorboard (relies on global_example_count being set properly)"""
    if not self.writer: return
    self.writer.add_scalar(tag, val, self.current_step)

  def update_step_count(self, batch_total):
    self.current_step += batch_total




  This file has been truncated. show original

  

  
    
    
  

  


and this works. I am not sure what I was doing wrong but I will investigate further if I have time. It seems like there are subtle differences in which process and where (before or after dist.init_process_group) you initialize your logger.