Feature request of Sparselty Gated Mixture-of-Experts and PowerNorm
# 🚀 Feature request

- The newer variant of MoE for Transformer as in [GShard: Scaling Giant Models with Conditional
Computation and Automatic Sharding](https://arxiv.org/abs/2006.16668) ([Relevant codes](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/research/moe.py))
- PowerNorm of [PowerNorm: Rethinking Batch Normalization in Transformers](https://arxiv.org/abs/2003.07845) ([Relevant codes](https://github.com/sIncerass/powernorm/blob/master/fairseq/modules/norms/mask_powernorm.py))

## Motivation

MoE as in GShard should be a crucial addition for this library, since the performance gain from it is rather significant as demonstrated in the paper. For example, Transformer+MoE achieved 44.3 avg BLEU on various NMT tasks with 10x less computes than with the baseline Transformer to get 36.9 avg BLEU. The codes for GShard has not been available yet, and if it will be published, it is most likely to be published in Tensorflow (or JAX) rather than PyTorch due to their use of TPUs. However, the codes for MoE does not seem to be complicated, so it must be easy to implement it.  

PowerNorm improved Wikitext-103 perplexity from 20.9 to 17.9 without additional computes simply by replacing LayerNorm with PowerNorm. Given that there is already PyTorch codes available, which is easy to transplant into this library, I think it's reasonable to suggest its feature request. 

## Your contribution
As for MoE, I'm currently writing my implementation of non-hierarchical MoE in PyTorch (no model parallelism as in GShard for numerous GPUs), and I'm going to compare its performance against the baseline. It should serve as a reference. My collaborator (@lucidrains) may write the same one with more flexibilities later, but no guarantee. You may want hierarchical MoE and GShard components as well. If you want them, my codes may not be as useful. 

As for PowerNorm, I don't think you need much help from me, as all you need is to replace LayerNorm with the above implemenation. I'll try to verify its performance gain, since replication of the results hasn't been done yet.  
Is there any update on this? Thanks.<|||||>@lucidrains made this [repo](https://github.com/lucidrains/mixture-of-experts) with me.<|||||>Thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
