Hi, rgwatwormhill. Sorry For I didn’t mention it clearly. Actually, I combine the MoE with the FFN module, following the design of Switch Transformer. This paper claimed this design can accelerate the training efficiency. Even though I haven’t observed in my experiments…