Hello everyone,
Has anyone experimented with Lamb optimizers in HF? I tried using https://github.com/cybertronai/pytorch-lamb 40 but I was only marginally able to increase batch size and the training loss curve was rather flat. If you’ve used lamb would you please share some tips. How did you initliaze it? I am not sure what to use in optimizer_grouped_parameters list of dictionaries that wrap model parameters. Also, I’ve seen some other people use a different lr scheduler with Lamb.
Thanks in advance.