Request to add Switch Transformer
Google has come up with yet another transformer: https://arxiv.org/pdf/2101.03961.pdf 
 Google released the source code for transformer-based mixture-of-experts (the switch architecture): https://github.com/tensorflow/mesh/blob/master/mesh_tensorflow/transformer/moe.py

According to https://www.infoq.com/news/2021/02/google-trillion-parameter-ai/ the model weights are not available yet.