Thanks for the response. That exactly what I thought initially, but then this snippet got me confused:
    def init_hidden(self, bsz):
    weight = next(self.parameters()).data
    if self.rnn_type == 'LSTM':
        return (Variable(weight.new(self.nlayers, bsz, self.nhid).zero_()),
                Variable(weight.new(self.nlayers, bsz, self.nhid).zero_()))
    else:
        return Variable(weight.new(self.nlayers, bsz, self.nhid).zero_())

From what I understand, self.parameters() returns all the model parameters, and  then gets the data from it.
But the part Variable(weight.new(self.nlayers, bsz, self.nhid).zero_()) seems to set all weights to zeros.
Also since the code is iterating over parameters, shouldn’t it be setting parameters on one layer at a time. But here it seems that all model parameters are set to zero at the same timem (self.nlayers, bsz, self.nhid)
I am sure I am missing some major link over here.