tom:

That might indicate that your learning rate is too high or your loss function is bad.


I can try and lower the learning rate, i have currently 0.01 with adam optimizer. I might simply drop MSELoss, but I was doing a hyper-parameter gridsearch test with Loss functions [CrossentropyLoss, MSELoss, MultiMarginLoss] and optimizers [Adam, Adadelta, RMSProp].
I thought it was possible to use MSELess for classifications tasks if the output of the model was Softmax like [0.1, 0.5, 0.1, 0.2, 0.1]and the y-tensor was like [0,1,0,0,0].
But i might be wrong? And since model outputs more like [0,0,0,0,1] results then the loss is constant and nothing is learned.
EDIT:
Okay lowered Lr by a factor of 10, that simply made it take 10 ish iterations before loss stayed constant.
However i lowered it by a factor of 100 and now loss looks like to be chaning properly!
Loss now:
tensor(0.1871, dtype=torch.float64, grad_fn=)
tensor(0.2140, dtype=torch.float64, grad_fn=)
tensor(0.1251, dtype=torch.float64, grad_fn=)
tensor(0.2270, dtype=torch.float64, grad_fn=)
tensor(0.1767, dtype=torch.float64, grad_fn=)
tensor(0.1708, dtype=torch.float64, grad_fn=)
tensor(0.0515, dtype=torch.float64, grad_fn=)
tensor(0.1856, dtype=torch.float64, grad_fn=)
tensor(0.0466, dtype=torch.float64, grad_fn=)
tensor(0.2470, dtype=torch.float64, grad_fn=)
tensor(0.1243, dtype=torch.float64, grad_fn=)
etc.
I will try and use this and simply add 1 couple of hundred more epochs and hope i don’t end up in an early local minimum