Sophia Optimizer


![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fe854c26e-dfc4-4c4a-a214-f5ec9b2d17be%2FUntitled.png?table=block&id=5b995fac-6f50-47cc-b301-57fae69b11bc&cache=v2)
![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F0e19cd19-6b44-49a8-bf8f-acb8ab16597c%2FUntitled.png?table=block&id=b20b53b2-8865-4296-8f62-8f7954747900&cache=v2)




> [Sophia: A Scalable Stochastic Second-order Optimizer for Language...](https://arxiv.org/abs/2305.14342)





