Hi! I was constructing a multilayer LSTM by stacking a bunch of LSTM in an list.
I found out that when I’m using the python list, the loss is weird.
But when I’m using the nn.ModuleList(), it’s just normal.
So why can’t I use the python list? Do I have to use nn.Modulelist()?
Here’s the code and loss when using nn.Modulelist()
class MultiLayerLSTM(nn.Module):
    def __init__(self, input_size, hidden_size,num_layers,batch_first):
        super(MultiLayerLSTM, self).__init__()
        self.num_layers = num_layers
        self.LSTMs = nn.ModuleList()
        self.LSTMs.append(NaiveLSTM(input_size, hidden_size, batch_first=True))
        for i in range(num_layers-1):
            self.LSTMs.append(NaiveLSTM(hidden_size, hidden_size, batch_first=True))

    def forward(self,x,state):
        (h0s,c0s) = state
        for i in range(self.num_layers):
            x,_ = self.LSTMs[i](x,(h0s[i].unsqueeze(0),c0s[i].unsqueeze(0)))
        return x,_

Epoch [1/2], Step [100/600], Loss: 0.6575
Epoch [1/2], Step [200/600], Loss: 0.4976
Epoch [1/2], Step [300/600], Loss: 0.1577
Epoch [1/2], Step [400/600], Loss: 0.2070
Epoch [1/2], Step [500/600], Loss: 0.0443
Epoch [1/2], Step [600/600], Loss: 0.1664
Epoch [2/2], Step [100/600], Loss: 0.0811
Epoch [2/2], Step [200/600], Loss: 0.1011
Epoch [2/2], Step [300/600], Loss: 0.1325
Epoch [2/2], Step [400/600], Loss: 0.1050
Epoch [2/2], Step [500/600], Loss: 0.1075
Epoch [2/2], Step [600/600], Loss: 0.0404

Here’s the code and loss when using python list:
class MultiLayerLSTM(nn.Module):
    def __init__(self, input_size, hidden_size,num_layers,batch_first):
        super(MultiLayerLSTM, self).__init__()
        self.num_layers = num_layers
        self.LSTMs = []
        self.LSTMs.append(NaiveLSTM(input_size, hidden_size, batch_first=True))
        for i in range(num_layers-1):
            self.LSTMs.append(NaiveLSTM(hidden_size, hidden_size, batch_first=True))

    def forward(self,x,state):
        (h0s,c0s) = state
        for i in range(self.num_layers):
            x,_ = self.LSTMs[i](x,(h0s[i].unsqueeze(0),c0s[i].unsqueeze(0)))
        return x,_

Epoch [1/2], Step [100/600], Loss: 2.2161
Epoch [1/2], Step [200/600], Loss: 2.1421
Epoch [1/2], Step [300/600], Loss: 2.1189
Epoch [1/2], Step [400/600], Loss: 2.0571
Epoch [1/2], Step [500/600], Loss: 2.1516
Epoch [1/2], Step [600/600], Loss: 2.0326
Epoch [2/2], Step [100/600], Loss: 1.9656
Epoch [2/2], Step [200/600], Loss: 1.9204
Epoch [2/2], Step [300/600], Loss: 1.9620
Epoch [2/2], Step [400/600], Loss: 1.8612
Epoch [2/2], Step [500/600], Loss: 1.8466
Epoch [2/2], Step [600/600], Loss: 1.8985