Body: <p>How do I interpret a large variance of a loss function?</p>
  
  <p>I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like this:
  <a href="https://i.sstatic.net/LvhRs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LvhRs.png" alt="a plot of the loss function"></a></p>
  
  <p>The green dots are the loss averaged over 100 epochs and the purple dots are the loss for each epoch.</p>
  
  <p>(You can ignore the missing part, I just did not save the loss values for these epochs)</p>
  
  <p>Is such a large variance a bad sign? And what are my options for tuning to get it to converge faster? Is the network to large or too small for my training data? Should I have a look at batch size?</p>
  
  <ul>
  <li>Learning rate parameter: 2.5e-4</li>
  <li>Training data size: 395 MB</li>
  </ul>
  
  <p>GPT-2 parameters:</p>
  
  <pre><code>{
    "n_vocab": 50000,
    "n_ctx": 1024,
    "n_embd": 768,
    "n_head": 12,
    "n_layer": 12
  }
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-08T18:07:34.750Z
Comments.Text: what is your learning rate/ dataset size
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-08T18:10:25.457Z
Comments.Text: @mshlis I added the parameters / learning rate / data size to the question. But I think the learning rate is changed by the optimizer over time, isn't it?
Comments.UserDisplayName: allo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-08T18:12:57.563Z
Comments.Text: no the optmiizer will add other terms not change the LR (an lr-scheduler will change the LR)... Also you are using their pretrained weights right? (did they also use a vocab of 50K?). Also you dont mean epoch, you mean iteration right?
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-08T18:17:13.290Z
Comments.Text: No, I started from scratch, because I want to train it with some german text. Their model has 50257 tokens vocabulary, the other parameters are like the their small (117M) model.
Comments.UserDisplayName: allo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-08T18:28:36.457Z
Comments.Text: in other words you are training a transformer encoder. (GPT2 is mostly defined by its actual training (data/weights) and how its transferable) -- so my guess this is understandable given that in most iterations it hasnt seen enough of the data
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-08T18:31:21.450Z
Comments.Text: Should I do something for tuning the process or do I just need to wait longer?
Comments.UserDisplayName: allo
Tags: <objective-functions><transformer><gpt>
Title: How to interpret a large variance of the loss function?
UserDisplayName: allo
ThreadText: # How to interpret a large variance of the loss function?
  
  (Asked by: username_0 on 2019-08-08)
  
  How do I interpret a large variance of a loss function?
  
  I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like this:
  [![a plot of the loss function](https://i.sstatic.net/LvhRs.png)](https://i.sstatic.net/LvhRs.png)
  
  The green dots are the loss averaged over 100 epochs and the purple dots are the loss for each epoch.
  
  (You can ignore the missing part, I just did not save the loss values for these epochs)
  
  Is such a large variance a bad sign? And what are my options for tuning to get it to converge faster? Is the network to large or too small for my training data? Should I have a look at batch size?
  
  * Learning rate parameter: 2.5e-4
  * Training data size: 395 MB
  
  GPT-2 parameters:
  
  ```
  {
    "n_vocab": 50000,
    "n_ctx": 1024,
    "n_embd": 768,
    "n_head": 12,
    "n_layer": 12
  }
  
  ```
  
  **Tags:** objective-functions, transformer, gpt
  
  > **Comments:**
  > **username_1**: what is your learning rate/ dataset size  
  > **username_0**: @username_1 I added the parameters / learning rate / data size to the question. But I think the learning rate is changed by the optimizer over time, isn't it?  
  > **username_1**: no the optmiizer will add other terms not change the LR (an lr-scheduler will change the LR)... Also you are using their pretrained weights right? (did they also use a vocab of 50K?). Also you dont mean epoch, you mean iteration right?  
  > **username_0**: No, I started from scratch, because I want to train it with some german text. Their model has 50257 tokens vocabulary, the other parameters are like the their small (117M) model.  
  > **username_1**: in other words you are training a transformer encoder. (GPT2 is mostly defined by its actual training (data/weights) and how its transferable) -- so my guess this is understandable given that in most iterations it hasnt seen enough of the data  
  > **username_0**: Should I do something for tuning the process or do I just need to wait longer?  
  
  ---
  
  ## 0 answers
  
  