Interesting new paper from Google improving upon T5.

  
      
      arXiv.org
  
  
    

Switch Transformers: Scaling to Trillion Parameter Models with Simple and... 24

In deep learning, models typically reuse the same parameters for all inputs.
Mixture of Experts (MoE) defies this and instead selects different parameters
for each incoming example. The result is a sparsely-activated model -- with
outrageous numbers...