Solved by mrshenli in post #27 
              
              
                Yep, because BatchNorm would trigger DDP comm in forward as well. In that case, need to move the signal checking before forward, but it will be slower. The following code should work. 
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn as nn
import torch…