Hi
can some one explain me the difference between:
Screenshot from 2020-11-29 17-22-501000×300 9.53 KB
On the left side Pytorch official implementation.
On the right side I took formulas for Wikipedia and A.Karpathy article.
Both of the formulas claims that they are Elman implementations. But in Wikipedia and  A.Karpathy article they have 3 set of weights W_i, W_h, W_y when in Pytorch implementation only 2 set of weights.
PS:
This answer does not explain anything.

  
    
    
    RNN implementation incorrect in PyTorch Tutorials? 
  
  
    Agree. When you check out simple RNN code result, it makes us wonder why the code in the RNN tutorial is different from the definition of RNN. 
import torch.nn as nn
rnn = nn.RNN(10, 10, batch_first=True)

inp = torch.randn((1,3,10))

result = rnn(inp)

print(result[0].shape)
print(result[1].shape)
print(result[0][0])
print(result[1])

Same result of : 
tensor([[[-0.0253,  0.5289, -0.2582, -0.5125, -0.4577,  0.5456, -0.4042, 
-0.6476, -0.4210, -0.5473]]], grad_fn=)
  


Sources:

https://pytorch.org/docs/stable/generated/torch.nn.RNN.html 1
https://en.wikipedia.org/wiki/Recurrent_neural_network
http://karpathy.github.io/2015/05/21/rnn-effectiveness/