Hello, I want to know how the MPI_Allreduce works in asynchronous mode when the gradients are calculated. Suppose we have 3 processes. If the first epoch is finished and only one process have update the gradients, when it takes the gradients from a shared buffer it takes NaN in the SUM of the process that havent finished ?? Im pretty lost here because Allreduce is a blocking primitive but the training doesnt stop for it.