I’ve implemented cosine normalization [1], attempting to match the paper [2].
However, when I use this code, I end up getting NaNs in _grad.data when calling .backward() after a few thousand samples presented to my network (exact count varies). Without using CosNorm (either with stock ResNet, or just switching to BatchNorm), the problem seems to go away (can’t prove a negative, though, since it might just be slower to happen).
My primary question is: How can I reliably investigate issues like this? I registered hooks, and nothing passed in there had NaNs, and so I eventually resorted to randomly disabling or changing parts of the code until I got enough clues to find a culprit (and even now, all the evidence I have is circumstantial).
My secondary question is: Any ideas why this normalization would be unstable? I was exclusively wrapping Conv2d instances.
Thanks!

https://gist.github.com/elistevens/5383d51c6c3b3f756ce3b312ef53f3a8 59

https://arxiv.org/pdf/1702.05870v2.pdf 22 .