Hello everyone, let me explain you a little background of my project and then I will tell you what problem I am facing so you get a clear picture of my problem.
so using pytroch.nn.RNN I trained neural network with 4 input neuron, 2 hidden layers , each have 8 neurons and 2 output neurons. so I trained my RNN model and I choose relu in 'nonlinearity ’ option and everything is fine there , my results are also ok. here is the picture of my model.

1777×673 84.6 KB

(its just half code because I am not allowed to post 2 pictures)
now as per my project requirements, I had to make same RNN structure from scratch using trained weights of the above-inbuilt model and I did that but my results are not matching when I apply relu activation function as ( relu(x) = max(0,x) ). but when I use ‘Tanh’ activation in inbuilt model and use that trained weights, then my model which I made from scratch is giving similar results as inbuilt model. i know there are several types of ‘relu’ activation functions and I tried a lot to find out what relu is exactly used by inbuilt model but I couldn’t find any answer so please please anyone can tell me exactly what relu RNN module is using so that I can also use the same and get results. It will be great help as I am in last semester of my masters and I am stuck in relu.
i also gone through it several times and I am not able to run torch._VF.rnn_relu. which is given in source code and I tried my best to make it run but libraries which are mentioned in source code , that I am not able to import in my google colab notebook even after installing all of them in colab.
It’s documented within the source file,
https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html#RNN 1
torch._VF.rnn_relu
can someone please help me in this… thank you in advance