Hey @Lewis_Liu, which part of the program is distributed? Since torch.distributed does not support Windows yet, I assume the working version of the program on Windows does not use distributed training?