You can call allreduce in the forward pass, but beware that if you have multiple of these layers, or other layers that need to call any torch.distributed functions, that the order they are called in needs to be identical across all workers. If you end up with any learnable parameters, consider the concerns I expressed on this PR adding sync batch norm 7.