There is a paper that does a similar thing:
  
      assets.amazon.science
  
  
    
learning-the-learning-rate-for-gradient-descent-by-gradient-descent.pdf 2

495.89 KB

  
  
    
    
  
  


Besides: It needs too much memory! and I guess the second learning-rate is really sensitive