I recently moved my regression model from Keras to Pytorch and I have been getting much worse results in Pytorch, to say the least. At first the model wasn’t even converging, it was just getting worse with each epoch(the std of the error kept growing both on the training set and the validation set). This was happening even though I had taken the utmost care to have the weight/bias initializations, optimizer and loss function parameters, learning rate and batch size exactly the same as I had them in Keras.
But then, as I was trying out different architectures to see what the problem was, I accidentally forgot to add the ReLU activation to one of them and my mistake somehow made the model converge! After realizing this I tried removing the ReLU activations from the other architectures as well and they all started converging.
Has anyone else been experiencing the same problem? Is there something wrong with ReLU in Pytorch?
Also, as a side note, I haven’t been able to find the source code for torch.relu(), if anyone knows where to find it and could share a link it would be very helpful. Thank you!