@Digvijay_mishra
Parameters
...
        num_layers – Number of recurrent layers. E.g., setting num_layers=2 would mean stacking two RNNs together to form a stacked RNN, with the second RNN taking in outputs of the first RNN and computing the final results. Default: 1

        nonlinearity – The non-linearity to use. Can be either 'tanh' or 'relu'. Default: 'tanh'
...

As per the documentaton, the default non linearity in the RNN module is tanh. This might be the reason why your model performs well with inbuilt model with tanh
Both tanh and relu have their own advantages and disadvantages, so at a general level you will have to understand your data and model and apply accordingly