I completely agree with your revision but on the other hand do you know whether the following code would init only the first hidden state or all hidden states in the RNN layer. I would like to init only the first hidden layer and have the rest of them be adjusted according to the learning process (so the require_grad=True I believe), but I’m afraid that I will end up initing all hidden states by using this statement
(I don’t provide here all the code)
Class Encoder
def __init__(self, rnn_type, ntoken, emb_dim, nhid, nlayers, dropout=0.5):

    hidden0 = torch.Tensor(ntoken, emb_dim) # size
    nn.init.uniform(hidden0)
    self.hidden0 = nn.Parameter(hidden0, requires_grad=False)

def forward(self, input, hidden):

    output, hidden = self.rnn(input, self.hidden0)

    output = self.drop(output)
    decoded1 = self.decoder1(output.view(output.size(0)*output.size(1), output.size(2)))
    decoded2 = self.decoder2(decoded1)

    return decoded2.view(output.size(0), output.size(1), decoded2.size(1)), hidden

Does this do what I would like?