(3)

This quality gain was corroborated in later work (Narang et al., 2021).

Root Mean Square Scale Parameters. Our second example is the scale parameter in root mean
square (RMS) normalization (Zhang and Sennrich, 2019). Within the Transformer, rather than call-
ing layers back-to-back, there is an internal structure (referred to as sublayer calls) which improve
gradient propagation and training dynamics. Our sublayer calls match that of Raffel et al. (2019)
and consist of: (1) RMS normalization, (2) layer call (e.g. Self Attention), (3) dropout (Srivastava
et al., 2014), (4) add residual (He et al., 2015). RMS normalization scales the input vector x ∈ Rd
element-wise per the root-mean-square. It then rescales the output element-wise by multiplying with
a learned scale parameter g.

(4)

xi(cid:113)
(cid:80)d

1
d

i=1 x2
i

· gi

yi =
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS