Algorithm Train CF Threshold Neg. Log. Perp. (↑)

Dense
Dense-L
Top-2
Top-2
Top-3
Top-3

—
—
3.0
3.0
3.0
3.0

—
—
0.2
0.05
0.2
0.05

-1.474
-1.384
-1.354
-1.356
-1.351
-1.349

Table 21: Performance of top-2 and top-3 routing with different thresholds. Top-3 routing does
slightly better with lower thresholds than top-2 routing.

H MESH LAYOUT FOR DATA, MODEL AND EXPERT PARALLELISM WITH

FEW EXPERTS
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS