ST-MOE: DESIGNING STABLE AND TRANSFERABLE

SPARSE EXPERT MODELS

Barret Zoph∗
Google Brain

Irwan Bello∗†
Google Brain

Sameer Kumar

Google

Jeff Dean

Google Research

Yanping Huang

Google Brain

William Fedus∗
Google Brain

Nan Du

Google Brain

Noam Shazeer†
Google Brain

ABSTRACT
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS