  The \texttt{MPI\_Allreduce} collective operation is a core kernel of many
parallel codebases, particularly for reductions over a single value per
process. The commonly used allreduce recursive-doubling algorithm obtains the
lower bound message count, yielding optimality for small reduction sizes based
on node-agnostic performance models. However, this algorithm yields duplicate
messages between sets of nodes. Node-aware optimizations in MPICH remove
duplicate messages through use of a single master process per node, yielding a
large number of inactive processes at each inter-node step. In this paper, we
present an algorithm that uses the multiple processes available per node to
reduce the maximum number of inter-node messages communicated by a single
process, improving the performance of allreduce operations, particularly for
small message sizes.

Distributed, Parallel, and Cluster Computing