Answers.Body: <p>It turns out that the zig-zag pattern is an inherent effect of using a word embedding layer. I don't fully understand the phenomenon, but I believe it has a strong correlation with the embeddings acting as a sort of <em>memory slots</em>, which can change relatively quickly, and the LSTM generating a summary of the sequence, so that the model can remember past combinations.</p>
  <p>I found this plot of a <a href="https://www.d2l.ai/chapter_natural-language-processing/word2vec-gluon.html" rel="nofollow noreferrer">training loss curve of word2vec</a> and it exhibits the same per-epoch pattern.</p>
  <p><a href="https://i.sstatic.net/ZtLAj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZtLAj.png" alt="word2vec loss" /></a></p>
  <h3>Edit</h3>
  <p>After conducting several experiments, I've isolated the causes. It seems that this is an indirect effect of having a large model capacity. In my case, I had too large word embeddings (size 1024) and too many classes (2002), which also increases model capacity, so the model was doing an almost per-sample learning. Reducing both resulted in a smooth-as-silk learning curve and a better generalisation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-30T12:02:23.020Z
Answers.LastActivityDate: 2019-08-01T11:23:30.283Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: David
Answers.Body: <p>This weird pattern can be caused by a big learning rate. Check this: <a href="https://stackoverflow.com/a/49095437/13164928">https://stackoverflow.com/a/49095437/13164928</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-07T09:39:02.040Z
Answers.LastActivityDate: 2024-04-07T09:39:02.040Z
Answers.UserDisplayName: ZappaBoy
Body: <p>I'm training a text classifier in PyTorch and I'm experiencing an unexplainable cyclical pattern in the loss curve. The loss drops drastically at the beginning of each epoch and then starts rising slowly. However, the global convergence pattern seems OK. Here's how it looks:</p>
  
  <p><a href="https://i.sstatic.net/15Azq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/15Azq.png" alt="loss curve (global)"></a><a href="https://i.sstatic.net/4VTjD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4VTjD.png" alt="loss curve (zoom)"></a></p>
  
  <p>The model is very basic and I'm using the Adam optimizer with default parameters and a learning rate of 0.001. Batches are of 512 samples. I've checked and tried a lot of stuff, so I'm running out of ideas, but I'm sure I've made a mistake somewhere.</p>
  
  <h1>Things I've made sure of:</h1>
  
  <ul>
  <li>Data is delivered correctly (VQA v1.0 questions).</li>
  <li>DataLoader is shuffling the dataset.</li>
  <li>LSTM's memory is being zeroed correctly</li>
  <li>Gradient isn't leaking through input tensors.</li>
  </ul>
  
  <h1>Things I've already tried:</h1>
  
  <ul>
  <li>Lowering the learning rate. Pattern remains, although amplitude is lower.</li>
  <li>Training without momentum (plain SGD). Gradient noise masks the pattern a bit, but it's still there.</li>
  <li>Using a smaller batch size (gradient noise can grow until it kinda masks the pattern, but that's not like solving it).</li>
  </ul>
  
  <h1>The model</h1>
  
  
  
  <pre><code>class QuestionAnswerer(nn.Module):
  
      def __init__(self):
          super(QuestionAnswerer, self).__init__()
          self._wemb = nn.Embedding(N_WORDS, HIDDEN_UNITS, padding_idx=NULL_ID)
          self._lstm = nn.LSTM(HIDDEN_UNITS, HIDDEN_UNITS)
          self._final = nn.Linear(HIDDEN_UNITS, N_ANSWERS)
  
      def forward(self, question, length):
          B = length.size(0)
          embed = self._wemb(question)
          hidden = self._lstm(embed)[0][length-1, torch.arange(B)]
          return self._final(hidden)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-27T00:30:41.387Z
Comments.Text: This looks an awful lot like you are plotting error at the end of each batch, rather then at the end of each epoch. I think you'll find if you plot just the epoch error, you'll get a nice curve. This is normal behaviour for a neural network if you simply plot each batch error, rather then epoch.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-27T11:09:38.163Z
Comments.Text: Yes, I do plot sub-epoch wise. My concern is not the niceness of the curve, but what the curve is telling me. And what it's telling me is that I'm missing something, either a cause for this behaviour or that I'm unadvertedly interferring on the training. Such an abbrupt fall of the loss is not common while using momentum. But you gave me an idea: I have a ton of output classes, so maybe this is a sign of the model memorizing samples. I'll test more on monday.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-28T10:36:31.877Z
Comments.Text: The cause for this behaviour is because you are plotting sub-epoch wise. This is meant to happen, it's normal. Try plotting *only* the first batch's error, and I'm sure you'll notice a nice curve. Then try *only* the second, and so on. Then, if you were to plot each of these curves sequentially, the graph in your question is what you would get.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-28T11:47:58.677Z
Comments.Text: Sorry, but that's not the cause. It's the reason for me being able to see it, and if I'd plot only once per epoch the thing would still be there, only I wouldn't notice it. Plotting sub-epoch wise is a good practice because it helps noticing coding errors, as well as analyzing the convergence and optimization nature of the net. So I guess you don't know why this pattern shows up and that's ok, me neither. I'll test more soon.
Comments.UserDisplayName: David
Tags: <long-short-term-memory><objective-functions><pytorch><learning-curve>
Title: LSTM text classifier shows unexpected cyclical pattern in loss
UserDisplayName: David
ThreadText: # LSTM text classifier shows unexpected cyclical pattern in loss
  
  (Asked by: username_0 on 2019-07-26)
  
  I'm training a text classifier in PyTorch and I'm experiencing an unexplainable cyclical pattern in the loss curve. The loss drops drastically at the beginning of each epoch and then starts rising slowly. However, the global convergence pattern seems OK. Here's how it looks:
  
  The model is very basic and I'm using the Adam optimizer with default parameters and a learning rate of 0.001. Batches are of 512 samples. I've checked and tried a lot of stuff, so I'm running out of ideas, but I'm sure I've made a mistake somewhere.
  Things I've made sure of:
  
  Data is delivered correctly (VQA v1.0 questions).
  DataLoader is shuffling the dataset.
  LSTM's memory is being zeroed correctly
  Gradient isn't leaking through input tensors.
  
  Things I've already tried:
  
  Lowering the learning rate. Pattern remains, although amplitude is lower.
  Training without momentum (plain SGD). Gradient noise masks the pattern a bit, but it's still there.
  Using a smaller batch size (gradient noise can grow until it kinda masks the pattern, but that's not like solving it).
  
  The model
  class QuestionAnswerer(nn.Module):
  
      def __init__(self):
          super(QuestionAnswerer, self).__init__()
          self._wemb = nn.Embedding(N_WORDS, HIDDEN_UNITS, padding_idx=NULL_ID)
          self._lstm = nn.LSTM(HIDDEN_UNITS, HIDDEN_UNITS)
          self._final = nn.Linear(HIDDEN_UNITS, N_ANSWERS)
  
      def forward(self, question, length):
          B = length.size(0)
          embed = self._wemb(question)
          hidden = self._lstm(embed)[0][length-1, torch.arange(B)]
          return self._final(hidden)
  
  **Tags:** long-short-term-memory, objective-functions, pytorch, learning-curve
  > **Comments:**
  > **username_1**: This looks an awful lot like you are plotting error at the end of each batch, rather then at the end of each epoch. I think you'll find if you plot just the epoch error, you'll get a nice curve. This is normal behaviour for a neural network if you simply plot each batch error, rather then epoch.  
  > **username_0**: Yes, I do plot sub-epoch wise. My concern is not the niceness of the curve, but what the curve is telling me. And what it's telling me is that I'm missing something, either a cause for this behaviour or that I'm unadvertedly interferring on the training. Such an abbrupt fall of the loss is not common while using momentum. But you gave me an idea: I have a ton of output classes, so maybe this is a sign of the model memorizing samples. I'll test more on monday.  
  > **username_1**: The cause for this behaviour is because you are plotting sub-epoch wise. This is meant to happen, it's normal. Try plotting *only* the first batch's error, and I'm sure you'll notice a nice curve. Then try *only* the second, and so on. Then, if you were to plot each of these curves sequentially, the graph in your question is what you would get.  
  > **username_0**: Sorry, but that's not the cause. It's the reason for me being able to see it, and if I'd plot only once per epoch the thing would still be there, only I wouldn't notice it. Plotting sub-epoch wise is a good practice because it helps noticing coding errors, as well as analyzing the convergence and optimization nature of the net. So I guess you don't know why this pattern shows up and that's ok, me neither. I'll test more soon.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2019-07-30. Score: 0 | Accepted answer
  
  It turns out that the zig-zag pattern is an inherent effect of using a word embedding layer. I don't fully understand the phenomenon, but I believe it has a strong correlation with the embeddings acting as a sort of *memory slots*, which can change relatively quickly, and the LSTM generating a summary of the sequence, so that the model can remember past combinations.
  
  I found this plot of a [training loss curve of word2vec](https://www.d2l.ai/chapter_natural-language-processing/word2vec-gluon.html) and it exhibits the same per-epoch pattern.
  
  [![word2vec loss](https://i.sstatic.net/ZtLAj.png)](https://i.sstatic.net/ZtLAj.png)
  
  ### Edit
  
  After conducting several experiments, I've isolated the causes. It seems that this is an indirect effect of having a large model capacity. In my case, I had too large word embeddings (size 1024) and too many classes (2002), which also increases model capacity, so the model was doing an almost per-sample learning. Reducing both resulted in a smooth-as-silk learning curve and a better generalisation.
  
  
  ---
  
  ### Answer by username_2 on 2024-04-07. Score: 1
  
  This weird pattern can be caused by a big learning rate. Check this: <https://stackoverflow.com/a/49095437/13164928>
  
  
  ---
  
  