Hi Thomas
To take you inform, I implemented by myself the LARS optimizer and tested it without any fine tuning yet. Using a lr of 0.01, clipping the trust ratio in [1e-4, 1], without momentum nor weight_decay, it already slightly outperform adam optimizer. I’ll continue with LARS for now, thank you for your suggestions. As I ended up using LARS I marked your last answer as the solution. I want to highlight also that using adam with amsgrad was more stable in my case than without amsgrad.
Best regards,
Thomas