our multiplicative interactions very efﬁcient). While we did note these methods to increase model
instabilities in Section 3.1, using the router z-loss in our models prevented any further instabilities.
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS