Optimization, Mathematics.
to me by George Nemes on math.SE. We can summarize it as follows:‌ ‌ 1) Loss initially drops sharply as negative log 2) Followed by slightly-super-exponential rate $1/(x e^x)$. 3) Eventually switching into traditional “exponential” decay. ‌ ‌ x-axis in this graph gives steps normalized by the