Answers.Body: <p>I tried to play with your code and found changing loss function to the cross_entropy alternate of negative log-likelihood makes the difference between 2000th epoch's loss and  9000th epoch's loss is greater about 0.2 alternate of 0.09</p>
  
  <p>I also tried to change optimizer and learning rate but no loss didn't improve.</p>
  
  <p>you can explore the <a href="https://gist.github.com/DevMahmoud10/4d17cc4eb5031b106836349130e9d842" rel="nofollow noreferrer">modified code</a> may help you with another idea</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-26T03:56:11.773Z
Answers.Text: Thank you for your reply! What I am confused about is why don't, for example, the first 2 entries (which should be 0 and 0) get updated. In your example (as well as my tests) they stay at 50% both (i.e. -6.9315e-01). Shouldn't they at least start to move towards the right value, as that would obviously reduce the loss function?
Answers.UserDisplayName: JohnDoe122
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-21T15:50:24.007Z
Answers.LastActivityDate: 2019-03-21T15:50:24.007Z
Answers.UserDisplayName: Mahmoud
Body: <p>I am writing my first LSTM network and I would really appreciate if someone can tell me if it is right (the loss seems to go down very slowly and before playing around with hyper parameters I want to make sure that the code is actually doing what I want). The code is meant to go through some time series and label each point according to some categories. In the version I am putting here there are just 2 categories: 0, if the value of the point is 1 and 1 otherwise (I know it’s a bit weird, but I didn’t choose the labels). So this is the code:</p>
  
  <pre><code>import torch
  import torch.nn as nn
  import torch.nn.functional as F
  from torch.autograd import Variable
  import torch.optim as optim
  import numpy as np
  from fastai.learner import *
  
  torch.manual_seed(1)
  torch.cuda.set_device(0)
  
  bs = 2
  
  x_trn = torch.tensor([[1.0000, 1.0000],
          [1.0000, 0.9870],
          [0.9962, 0.9848],
          [1.0000, 1.0000]]).cuda()
  
  y_trn = torch.tensor([[0, 0],
          [0, 1],
          [1, 1],
          [0, 0]]).cuda()
  
  n_hidden = 5
  n_classes = 2
  
  class TESS_LSTM(nn.Module):
      def __init__(self, nl):
          super().__init__()
          self.nl = nl
          self.rnn = nn.LSTM(1, n_hidden, nl)
          self.l_out = nn.Linear(n_hidden, n_classes)
          self.init_hidden(bs)
  
      def forward(self, input):
          outp,h = self.rnn(input.view(len(input), bs, -1), self.h)
          return F.log_softmax(self.l_out(outp),dim=1)
  
      def init_hidden(self, bs):
          self.h = (V(torch.zeros(self.nl, bs, n_hidden)),
                    V(torch.zeros(self.nl, bs, n_hidden)))
  
  model = TESS_LSTM(1).cuda()
  
  loss_function = nn.NLLLoss()
  
  optimizer = optim.Adam(model.parameters(), lr=0.01)
  
  for epoch in range(10000):  
      model.zero_grad()
      tag_scores = model(x_trn)
      loss = loss_function(tag_scores.reshape(4*bs,n_classes), y_trn.reshape(4*bs))
      loss.backward()
      optimizer.step()
  
      if epoch%1000==0:
          print("Loss at epoch %d = " %epoch, loss)
  
  print(model(x_trn), y_trn)
  </code></pre>
  
  <p>The (super reduced in size) time series should be <code>[1,1, 0.9962,1]</code>, with labels <code>[0,0,1,0]</code> and <code>[1, 0.9870, 0.9848,1]</code> with labels <code>[0,1,1,0]</code> and the batch size should be 2. I really hope I didn’t mess up the dimensionalities, but I tried to make it in a shape accepted by the LSTM. This is the output:</p>
  
  <pre><code>Loss at epoch 0 = tensor(1.3929, device='cuda:0', grad_fn=&amp;lt;NllLossBackward&amp;gt;)
  Loss at epoch 1000 = tensor(0.8939, device='cuda:0', grad_fn=&amp;lt;NllLossBackward&amp;gt;) 
  Loss at epoch 2000 = tensor(0.8664, device='cuda:0', grad_fn=&amp;lt;NllLossBackward&amp;gt;) 
  Loss at epoch 3000 = tensor(0.8390, device='cuda:0', grad_fn=&amp;lt;NllLossBackward&amp;gt;) 
  Loss at epoch 4000 = tensor(0.8339, device='cuda:0', grad_fn=&amp;lt;NllLossBackward&amp;gt;) 
  Loss at epoch 5000 = tensor(0.8288, device='cuda:0', grad_fn=&amp;lt;NllLossBackward&amp;gt;) 
  Loss at epoch 6000 = tensor(0.8246, device='cuda:0', grad_fn=&amp;lt;NllLossBackward&amp;gt;) 
  Loss at epoch 7000 = tensor(0.8202, device='cuda:0', grad_fn=&amp;lt;NllLossBackward&amp;gt;) 
  Loss at epoch 8000 = tensor(0.8143, device='cuda:0', grad_fn=&amp;lt;NllLossBackward&amp;gt;)
  Loss at epoch 9000 = tensor(0.8108, device='cuda:0', grad_fn=&amp;lt;NllLossBackward&amp;gt;)
  
  (tensor([[[-9.0142e-01, -1.2631e+01],
            [-9.3762e-01, -9.6707e+00]],
  
           [[-1.3467e+00, -3.9542e+00],
            [-2.2005e+00, -7.6977e-01]],
  
           [[-2.4500e+01, -1.9363e-02],
            [-2.3349e+01, -6.2210e-01]],
  
           [[-1.0969e+00, -2.1953e+01],
            [-6.9776e-01, -1.8608e+01]]], device='cuda:0',
          grad_fn=&lt;LogSoftmaxBackward&gt;), tensor([[0, 0],
           [0, 1],
           [1, 1],
           [0, 0]], device='cuda:0'))
  </code></pre>
  
  <p>The loss doesn’t go down too fast (I expected it to overfit and go really close to zero). The actual values are okish (the smaller one is always the right one), but they can be definitely improved. Can someone tell me if my code is doing what I want (and maybe suggest why is the loss still big - maybe I need a smaller LR?)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-20T20:02:51.420Z
Comments.Text: Welcome to SE:AI!  This looks like it might be a better fit for Data Science.  (Let me know if you'd like me to migrate.)
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-20T23:17:50.193Z
Comments.Text: @DukeZhou sure, if you think that is better. Thank you!
Comments.UserDisplayName: JohnDoe122
Tags: <long-short-term-memory><convergence><pytorch>
Title: LSTM is not converging
UserDisplayName: JohnDoe122
ThreadText: # LSTM is not converging
  
  (Asked by: username_0 on 2019-03-20)
  
  I am writing my first LSTM network and I would really appreciate if someone can tell me if it is right (the loss seems to go down very slowly and before playing around with hyper parameters I want to make sure that the code is actually doing what I want). The code is meant to go through some time series and label each point according to some categories. In the version I am putting here there are just 2 categories: 0, if the value of the point is 1 and 1 otherwise (I know it’s a bit weird, but I didn’t choose the labels). So this is the code:
  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  from torch.autograd import Variable
  import torch.optim as optim
  import numpy as np
  from fastai.learner import *
  
  torch.manual_seed(1)
  torch.cuda.set_device(0)
  
  bs = 2
  
  x_trn = torch.tensor([[1.0000, 1.0000],
          [1.0000, 0.9870],
          [0.9962, 0.9848],
          [1.0000, 1.0000]]).cuda()
  
  y_trn = torch.tensor([[0, 0],
          [0, 1],
          [1, 1],
          [0, 0]]).cuda()
  
  n_hidden = 5
  n_classes = 2
  
  class TESS_LSTM(nn.Module):
      def __init__(self, nl):
          super().__init__()
          self.nl = nl
          self.rnn = nn.LSTM(1, n_hidden, nl)
          self.l_out = nn.Linear(n_hidden, n_classes)
          self.init_hidden(bs)
  
      def forward(self, input):
          outp,h = self.rnn(input.view(len(input), bs, -1), self.h)
          return F.log_softmax(self.l_out(outp),dim=1)
  
      def init_hidden(self, bs):
          self.h = (V(torch.zeros(self.nl, bs, n_hidden)),
                    V(torch.zeros(self.nl, bs, n_hidden)))
  
  model = TESS_LSTM(1).cuda()
  
  loss_function = nn.NLLLoss()
  
  optimizer = optim.Adam(model.parameters(), lr=0.01)
  
  for epoch in range(10000):  
      model.zero_grad()
      tag_scores = model(x_trn)
      loss = loss_function(tag_scores.reshape(4*bs,n_classes), y_trn.reshape(4*bs))
      loss.backward()
      optimizer.step()
  
      if epoch%1000==0:
          print("Loss at epoch %d = " %epoch, loss)
  
  print(model(x_trn), y_trn)
  
  The (super reduced in size) time series should be [1,1, 0.9962,1], with labels [0,0,1,0] and [1, 0.9870, 0.9848,1] with labels [0,1,1,0] and the batch size should be 2. I really hope I didn’t mess up the dimensionalities, but I tried to make it in a shape accepted by the LSTM. This is the output:
  Loss at epoch 0 = tensor(1.3929, device='cuda:0', grad_fn=<NllLossBackward>)
  Loss at epoch 1000 = tensor(0.8939, device='cuda:0', grad_fn=<NllLossBackward>) 
  Loss at epoch 2000 = tensor(0.8664, device='cuda:0', grad_fn=<NllLossBackward>) 
  Loss at epoch 3000 = tensor(0.8390, device='cuda:0', grad_fn=<NllLossBackward>) 
  Loss at epoch 4000 = tensor(0.8339, device='cuda:0', grad_fn=<NllLossBackward>) 
  Loss at epoch 5000 = tensor(0.8288, device='cuda:0', grad_fn=<NllLossBackward>) 
  Loss at epoch 6000 = tensor(0.8246, device='cuda:0', grad_fn=<NllLossBackward>) 
  Loss at epoch 7000 = tensor(0.8202, device='cuda:0', grad_fn=<NllLossBackward>) 
  Loss at epoch 8000 = tensor(0.8143, device='cuda:0', grad_fn=<NllLossBackward>)
  Loss at epoch 9000 = tensor(0.8108, device='cuda:0', grad_fn=<NllLossBackward>)
  
  (tensor([[[-9.0142e-01, -1.2631e+01],
            [-9.3762e-01, -9.6707e+00]],
  
           [[-1.3467e+00, -3.9542e+00],
            [-2.2005e+00, -7.6977e-01]],
  
           [[-2.4500e+01, -1.9363e-02],
            [-2.3349e+01, -6.2210e-01]],
  
           [[-1.0969e+00, -2.1953e+01],
            [-6.9776e-01, -1.8608e+01]]], device='cuda:0',
          grad_fn=), tensor([[0, 0],
           [0, 1],
           [1, 1],
           [0, 0]], device='cuda:0'))
  
  The loss doesn’t go down too fast (I expected it to overfit and go really close to zero). The actual values are okish (the smaller one is always the right one), but they can be definitely improved. Can someone tell me if my code is doing what I want (and maybe suggest why is the loss still big - maybe I need a smaller LR?)
  
  **Tags:** long-short-term-memory, convergence, pytorch
  > **Comments:**
  > **username_1**: Welcome to SE:AI!  This looks like it might be a better fit for Data Science.  (Let me know if you'd like me to migrate.)  
  > **username_0**: @DukeZhou sure, if you think that is better. Thank you!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-03-21. Score: 1
  
  I tried to play with your code and found changing loss function to the cross\_entropy alternate of negative log-likelihood makes the difference between 2000th epoch's loss and 9000th epoch's loss is greater about 0.2 alternate of 0.09
  
  I also tried to change optimizer and learning rate but no loss didn't improve.
  
  you can explore the [modified code](https://gist.github.com/DevMahmoud10/4d17cc4eb5031b106836349130e9d842) may help you with another idea
  
  > **Comments:**
  > **username_0**: Thank you for your reply! What I am confused about is why don't, for example, the first 2 entries (which should be 0 and 0) get updated. In your example (as well as my tests) they stay at 50% both (i.e. -6.9315e-01). Shouldn't they at least start to move towards the right value, as that would obviously reduce the loss function?  
  
  ---
  
  