  We examine natural expectations on communication performance using MPI
derived datatypes in comparison to the baseline, "raw" performance of
communicating simple, non-contiguous data layouts. We show that common MPI
libraries sometimes violate these datatype performance expectations, and
discuss reasons why this happens, but also show cases where MPI libraries
perform well. Our findings are in many ways surprising and disappointing.
First, the performance of derived datatypes is sometimes worse than the
semantically equivalent packing and unpacking using the corresponding MPI
functionality. Second, the communication performance equivalence stated in the
MPI standard between a single contiguous datatype and the repetition of its
constituent datatype does not hold universally. Third, the heuristics that are
typically employed by MPI libraries at type-commit time are insufficient to
enforce natural performance guidelines, and better type normalization
heuristics may have a significant performance impact. We show cases where all
the MPI type constructors are necessary to achieve the expected performance for
certain data layouts. We describe our benchmarking approach to verify the
datatype performance guidelines, and present extensive verification results for
different MPI libraries.

Distributed, Parallel, and Cluster Computing