I suggest to take a look at some strong benchmark protocols like ICLR 2021:

  
      

      github.com
  

  
    

SirRob1997/Crowded-Valley---Results 10


  This repository contains the results for the paper: "Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers"