Thanks for the reply! After reading over the PR for synced batch norm (really excited to see this functionality being baked into PyTorch btw, I think it’s a must have for proper distributed training), you seem to point out a potential deadlock when everything is running on the same process group. Is this a factor for my layer since no learnable parameters are used? I can always spin up a new process group for this layer to use if so.