@Digvijay_mishra The code looks good
Just compare the results of out_h2 between your implementation and inbuilt model. And use a common seed.
torch.manual_seed(42)

RNN by default uses tanh and if you mention ‘relu’ then it uses the default Relu