for future work.

12

65k262k1MBatch Size81828384858687SuperGLUE ScoreDenseSparse1e-45e-41e-3Learning Rate81828384858687SuperGLUE ScoreDenseSparse4.5

INSERTING SENTINELS TOKENS DURING FINE-TUNING

Sentinel tokens denote masked sequences in the span-corruption objective (Fedus et al., 2018; Devlin
et al., 2018). This differs from any ﬁne-tuning task we would likely encounter, leading to a domain
mismatch between pre-training and ﬁne-tuning. Table 6 illustrates the difference. We examine
whether modifying the ﬁne-tuning task to look more like the pre-training task effects results.

Objective
Span Corruption
Fine-Tuning
Fine-Tuning + Sentinels What is the capital of Illinois ? <X>

I like <X> the pool <Y> day .
What is the capital of Illinois ?

Inputs

Targets

<X> going to <Y> on a sunny

Springﬁeld

<X> Springﬁeld
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS