33

Algorithm Train CF Eval CF Neg. Log. Perp. (↑)

Dense
Dense-L
BPR Top-1
BPR Top-1

BPR Top-1
BPR Top-1

BPR Top-1
BPR Top-1

Top-1
Top-1
Top-2
Top-2

Top-1
Top-1
Top-2
Top-2

Top-1
Top-1
Top-2
Top-2

BPR Top-1
BPR Top-1

—
—
0.5
0.5
0.75
0.75
0.75
0.75
0.75
0.75
1.0
1.0
1.0
1.0
1.0
1.0
1.25
1.25
1.25
1.25
1.25
1.25

—
—
0.5
2.0
0.75
2.0
0.75
2.0
0.75
2.0
1.0
2.0
1.0
2.0
1.0
2.0
1.25
2.0
1.25
2.0
1.25
2.0

-1.474
-1.384
-1.433
-1.416
-1.428
-1.404
-1.424
-1.402
-1.409
-1.397
-1.397
-1.384
-1.392
-1.378
-1.386
-1.379
-1.378
-1.373
-1.375
-1.369
-1.376
-1.375

Table 18: Batch Prioritized Top-1 Routing (BPR) performance. BPR top-1 routing improves
quality when capacity factors are ≤ 1. However, once the capacity factor reaches 1.25, the improve-
ments greatly diminish and it underperforms top-2 routing. Future work can try BPR with top-2
routing, which should hopefully further improve the performance.

E PRE-TRAINING DATASET DETAILS
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS