You are right, the loss is suddenly growing. But it starts properly:
The sequence is: 1600, 1550, 1520, 600000, 5e17, nan
I am wondering why it suddenly goes up like that.
If I clip the loss, wont my gradient be 0 and training collapse?
Sigmoid seems an interesting idea, I will try that, thanks.