divyesh_rajpura:

Many people have implemented this in PyTorch but I seen all of them have just used base RNN models in both Encoder and Decoder which actually needs internal modification.


Posting links here would make responding easier.
Bahdanau’s formulation doesn’t seem to require any modifications to the built in RNN.
From the equations in Section A (Appendix) of the first paper, you could possibly write an inefficient(?) implementation using some primitives. Check the following pseudocode:
class EncoderRNNCell(nn.Module):
    def __init__(self, args):
        def _linear():
            return nn.Linear(args.input_size, args.hidden_size)
        self.We = _linear()
        self.Ue = _linear()

        self.Wz = _linear()
        self.Uz = _linear()

        self.Wr = _linear()
        self.Ur = _linear()

    def forward(self, xs):
        # xs: T x args.input_size
        T, H = x.size()
        # TODO: Init h, r with legit values

        hs = [h]
        for i in range(T):
            x = xs[i, :]
            h_ = F.tanh(self.We(x) + self.Ue(r*h))
            z = F.sigmoid(self.Wz(x) + self.Uz(h))
            r = F.sigmoid(self.Wr(x) + self.Ur(h))
            h = z*h + r*h_
            hs.append(h)

        hiddens = torch.stack(hs, dim=...)
        return hiddens


You can use hiddens[-1] as  h<T>.