Thank you!


How did you make this diagram?


From the graph, it seems like the weights receive two sets of gradients: one normal, and one that goes through max/abs ops, so I’m guessing it gets applied only to the largest weight? Very interesting indeed.


I did use L2 regularization for both SGD and AdamW. I tried different values, and it doesn’t seem to be very sensitive to it, but some values are better than others.