I’m training on an easy LSTM classifier for a 3-class classification task. I wasn’t expecting any of these issues, and I could find where I got wrong in my code.
I’m currently using pytorch built-in Embedding for the pre-processed one-hot embedding. And the rest should be quite straightforward.
import torch.nn as nn
import torch.autograd as autograd
import torch.nn.functional as F
import torch.optim as optim

vocab_size = len(vocab_ids) + 1
embedding_dim = 100
hidden_dim = 128
num_layers = 1

class lstm_sentiment(torch.nn.Module):
    
    def __init__(self) :
        super(lstm_sentiment, self).__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers = num_layers, batch_first=True)
        self.dropout = nn.Dropout(0.2)
        self.linear = nn.Linear(hidden_dim, 3)
        self.softmax = nn.LogSoftmax(dim=1)
        
    def forward(self, x, hidden):
        embeds = self.embedding(x)
        output, hidden = self.lstm(embeds, hidden)
        output = output[:,-1,:]
        output = self.dropout(output)
        output = self.linear(output)
        output = F.relu(output)
        output = self.softmax(output)
        return output, hidden
    
    def init_hidden(self):
        hidden = (torch.zeros(num_layers, batch_size, hidden_dim).cuda(), 
                  torch.zeros(num_layers, batch_size, hidden_dim).cuda())
        return hidden

There are several places that I’m not quite sure.

In LSTM there needs to be a hidden layer, which we can initialize it by either using random numbers or using all zeros. But I am struggling with the place where I call init_hidden() method. In a pytorch official tutorial which used LSTM, init_hidden() was called inside forward(self, x) function. But I have different opinion. In my understanding, if I return the hidden layer outputted by LSTM from forward(self, x, hidden) to my training process, the next batch in the same epoch will continue using the hidden states outputted from last batch, instead of initializing a new hidden layer with all zeros or all random numbers. I think it won’t benefit the training if initializing a new hidden layer every time entering forward(self, x) method. I’m not sure if I’m correct.
When I’m calling model.init_hidden() in training process, it seems I had to call  hidden = tuple([each.data for each in hidden]), which I’m not quite sure why.
My whole dataset is of size 100,000 but now I am only using 10,000 to speed it up. And I don’t think it would influence the result that much.

lr = 0.05
criterion = nn.NLLLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
epochs = 50
print_every = 1000
clip = 2  # gradient clipping

model = lstm_sentiment()
model.cuda()
model.train()

training_losses = []

for epoch in range(epochs):
    temp_loss = []
    hidden = model.init_hidden()
    for step, batch in enumerate(train_dataloader):
        optimizer.zero_grad()
        inputs = batch[0].cuda()
        labels = batch[1].cuda()
        hidden = tuple([each.data for each in hidden])
        output, hidden = model(inputs, hidden)
        loss = criterion(output, labels)
        loss.backward()
        temp_loss.append(loss.detach().cpu().numpy())
        nn.utils.clip_grad_norm_(model.parameters(), clip)
        optimizer.step()
    training_losses.append(sum(temp_loss) / len(temp_loss))
    print("Epoch: {}/{}...".format(epoch + 1, epochs),
         "Loss: {:.6f}...".format(sum(temp_loss) / len(temp_loss)))

Here’s the output result:
Epoch: 1/50... Loss: 1.098612...
Epoch: 2/50... Loss: 1.098612...
Epoch: 3/50... Loss: 1.098612...
Epoch: 4/50... Loss: 1.098612...
Epoch: 5/50... Loss: 1.098612...
Epoch: 6/50... Loss: 1.098612...
Epoch: 7/50... Loss: 1.098612...
Epoch: 8/50... Loss: 1.098613...
Epoch: 9/50... Loss: 1.104512...
Epoch: 10/50... Loss: 1.098612...
Epoch: 11/50... Loss: 1.098612...
Epoch: 12/50... Loss: 1.098612...
Epoch: 13/50... Loss: 1.098612...
Epoch: 14/50... Loss: 1.098612...
Epoch: 15/50... Loss: 1.098612...
Epoch: 16/50... Loss: 1.098612...
Epoch: 17/50... Loss: 1.098612...
Epoch: 18/50... Loss: 1.098612...
Epoch: 19/50... Loss: 1.098612...
Epoch: 20/50... Loss: 1.098612...
Epoch: 21/50... Loss: 1.098612...
Epoch: 22/50... Loss: 1.098612...
Epoch: 23/50... Loss: 1.098612...
Epoch: 24/50... Loss: 1.098612...
Epoch: 25/50... Loss: 1.098612...
Epoch: 26/50... Loss: 1.098612...
Epoch: 27/50... Loss: 1.098612...
Epoch: 28/50... Loss: 1.098612...
Epoch: 29/50... Loss: 1.098612...
Epoch: 30/50... Loss: 1.098612...
Epoch: 31/50... Loss: 1.098612...
Epoch: 32/50... Loss: 1.098612...
Epoch: 33/50... Loss: 1.098612...
Epoch: 34/50... Loss: 1.098612...
Epoch: 35/50... Loss: 1.098612...
Epoch: 36/50... Loss: 1.098612...
Epoch: 37/50... Loss: 1.098612...
Epoch: 38/50... Loss: 1.098612...
Epoch: 39/50... Loss: 1.098612...
Epoch: 40/50... Loss: 1.098612...
Epoch: 41/50... Loss: 1.098612...
Epoch: 42/50... Loss: 1.098612...
Epoch: 43/50... Loss: 1.098612...
Epoch: 44/50... Loss: 1.098612...
Epoch: 45/50... Loss: 1.098612...
Epoch: 46/50... Loss: 1.098612...
Epoch: 47/50... Loss: 1.098612...
Epoch: 48/50... Loss: 1.098612...
Epoch: 49/50... Loss: 1.098612...
Epoch: 50/50... Loss: 1.098612...

It turns out that the loss doesn’t change. I thought I fully understood LSTM but it seems I’m wrong. Can anyone help me please. PyTorch is really much better than tensorflow, which I don’t want to give it up.