Hi ezio98.
I can’t answer your question, but I’m a bit confused.   From what I have read about the MoE layer, the point of it is to facilitate the use (Mixture) of many different models (Experts) concurrently, but you say you are using the MoE layer on top of a single BertForMaskedLM model.
What are you hoping the MoE layer will do for you?   Does it have some other advantages?