This may not be exactly what you’re looking for, but this paper explores the performance impacts of different configurations of decoder, attention heads, hidden layers and intermediate layers.

  
      arxiv.org
  
  
    
2010.10499.pdf 5

552.32 KB