I am trying to implement a CNN for regression on images in PyTorch. I have a working model already implemented in Keras and I would like to translate it in PyTorch, but I am facing many issues.
Essentially, in Keras the model converges, whereas in PyTorch it doesn’t. In PyTorch I always have a constant training loss and the trained model always outputs the same value for any image. I initialized all the layers as in the working Keras model, I added l2 regularization as in Keras, I also implemented the same learning rate decay. Everything looks exactly the same, but in PyTorch my model doesn’t converge.
Is it possible that, with everything initialized the same way, a model does not converge in PyTorch while converging in Keras? If so, what would you suggest me to do in my specific case (constant training loss after one epoch and constant predictions)?
I have already tried to clip gradients and change learning rate, at the beginning I used Adam with lr = 0.001, then I tried 0.1 and 0.0001, always emulating Keras time-based decay.
Thank you in advance