divide the number of data sequences and the number of columns must evenly divide the model
dimensions being partitioned.
But if we have fewer than d experts then this layout will not work. To allow for fewer experts than
data parallelism rows in our mesh, we factorize the data dimension into two new dimensions: inner
(i) and outer (o) where i x o = d and the number of experts equals i. This transforms the logical 2D
mesh of shape d x m into a 3D mesh of shape o x i x m. See Figure 8 for a visualization of both
meshes 12.

I NOTE ON COMMUNICATION COSTS FOR DISTRIBUTED MODELS
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS