@sgugger  The model is the routing transformer language model (RoutingTransformerLM). The source code is here:
  
      
      github.com
  
  
    

lucidrains/routing-transformer 18


  Fully featured implementation of Routing Transformer