Solved by tom in post #6 
              
              
                First, I have to say you know a lot more about your situation also in training than me, so take all this with a grain of salt. 
The key advantage of LAMB/LARS to me here is that they work at the tensor level while Adam works at the “single scalar entry” level for the adaptiveness. LARS might a more …