I am training a custom patch-based network (4 layers) and I realized that, unless I set the lr to 0.0000001 to start converging somehow.
I feel like something is wrong with such a tuning.
However, if I run the training with lr = 0.01, loss will get huge, like several billions and eventually NaN.
Can this be related to initialisation ?