<p>How do I interpret a large variance of a loss function?</p>

<p>I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like this:
<a href="https://i.sstatic.net/LvhRs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LvhRs.png" alt="a plot of the loss function"></a></p>

<p>The green dots are the loss averaged over 100 epochs and the purple dots are the loss for each epoch.</p>

<p>(You can ignore the missing part, I just did not save the loss values for these epochs)</p>

<p>Is such a large variance a bad sign? And what are my options for tuning to get it to converge faster? Is the network to large or too small for my training data? Should I have a look at batch size?</p>

<ul>
<li>Learning rate parameter: 2.5e-4</li>
<li>Training data size: 395 MB</li>
</ul>

<p>GPT-2 parameters:</p>

<pre><code>{
  "n_vocab": 50000,
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12
}
</code></pre>

CC BY-SA 4.0
2019-08-08T18:07:34.750Z
what is your learning rate/ dataset size
mshlis
CC BY-SA 4.0
2019-08-08T18:10:25.457Z
@mshlis I added the parameters / learning rate / data size to the question. But I think the learning rate is changed by the optimizer over time, isn't it?
allo
CC BY-SA 4.0
2019-08-08T18:12:57.563Z
no the optmiizer will add other terms not change the LR (an lr-scheduler will change the LR)... Also you are using their pretrained weights right? (did they also use a vocab of 50K?). Also you dont mean epoch, you mean iteration right?
mshlis
CC BY-SA 4.0
2019-08-08T18:17:13.290Z
No, I started from scratch, because I want to train it with some german text. Their model has 50257 tokens vocabulary, the other parameters are like the their small (117M) model.
allo
CC BY-SA 4.0
2019-08-08T18:28:36.457Z
in other words you are training a transformer encoder. (GPT2 is mostly defined by its actual training (data/weights) and how its transferable) -- so my guess this is understandable given that in most iterations it hasnt seen enough of the data
mshlis
CC BY-SA 4.0
2019-08-08T18:31:21.450Z
Should I do something for tuning the process or do I just need to wait longer?
allo
<objective-functions><transformer><gpt>
How to interpret a large variance of the loss function?
allo
# How to interpret a large variance of the loss function?

(Asked by: username_0 on 2019-08-08)

How do I interpret a large variance of a loss function?
I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like this:

The green dots are the loss averaged over 100 epochs and the purple dots are the loss for each epoch.
(You can ignore the missing part, I just did not save the loss values for these epochs)
Is such a large variance a bad sign? And what are my options for tuning to get it to converge faster? Is the network to large or too small for my training data? Should I have a look at batch size?

Learning rate parameter: 2.5e-4
Training data size: 395 MB

GPT-2 parameters:
{
  "n_vocab": 50000,
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12
}

**Tags:** objective-functions, transformer, gpt

> **Comments:**
> **username_1**: what is your learning rate/ dataset size  
> **username_0**: @username_1 I added the parameters / learning rate / data size to the question. But I think the learning rate is changed by the optimizer over time, isn't it?  
> **username_1**: no the optmiizer will add other terms not change the LR (an lr-scheduler will change the LR)... Also you are using their pretrained weights right? (did they also use a vocab of 50K?). Also you dont mean epoch, you mean iteration right?  
> **username_0**: No, I started from scratch, because I want to train it with some german text. Their model has 50257 tokens vocabulary, the other parameters are like the their small (117M) model.  
> **username_1**: in other words you are training a transformer encoder. (GPT2 is mostly defined by its actual training (data/weights) and how its transferable) -- so my guess this is understandable given that in most iterations it hasnt seen enough of the data  
> **username_0**: Should I do something for tuning the process or do I just need to wait longer?  

---

## 0 answers

