I am trying to implement a CNN for regression on images in PyTorch. I have a working model already implemented in Keras and I would like to translate it in PyTorch, but I am facing many issues.
Essentially, in Keras the model converges, whereas in PyTorch it doesn’t.
In PyTorch I always obtain a constant training loss and the trained model always outputs the same value for any image, regardless of learning rate, batch size and apparently also for slightly different initializations, it always converges to the same loss: 0.0745.
I initialized all the layers as in the working Keras model, I added l2 regularization as in Keras, I also implemented the same learning rate decay. Everything looks exactly the same, but in PyTorch my model doesn’t converge.
After some debugging I found out that the last relu of my trained model is dead. This happens only to the last one, previous to that one all of the others seems to be working. How would you suggest to solve it? I was thinking of trying leaky relu/initialization of biases of conv layers to 0.1/directly removing the last relu.