Adding explicit expert positional information. We experimented with adding explicit positional
information into the outputs of the expert layer. We wanted to see if it either improved performance
or sped up convergence during the beginning of training when expert layers were drastically chang-
ing. We did this through adding an embedding corresponding to what expert each token was sent
(including an embedding if the token was dropped), but this did not improve performance.
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS