Answers.Body: <p>You can use the <a href="https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average" rel="nofollow noreferrer">Exponential Moving Average</a> method. This method is used in tensorbaord as a way to smoothen a loss curve plot.
  <a href="https://i.sstatic.net/b3xGj.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/b3xGj.jpg" alt="enter image description here"></a>
  The algorithm is as follow:
  <a href="https://i.sstatic.net/ACJHA.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ACJHA.jpg" alt="enter image description here"></a>
  However there is a small problem doing it this way. As you can see S_t is initialized with the starting value, which makes the starting curve inaccurate. 
  <a href="https://i.sstatic.net/G3V2D.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/G3V2D.png" alt="enter image description here"></a>
  The green curve is the ideal curve for the algorithm, but the purple curve is the predicted curve. The curve is not correct on the start. To solve this, a correction factor is added in, thus making the algorithm this:
  <a href="https://i.sstatic.net/uRDa6.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uRDa6.jpg" alt="enter image description here"></a>
  This introduces WeightedCount which decreases over time to 0.</p>
  
  <p>Exponential Moving Average is also used is other areas of deep learning, the most notable being some optimization algorithms. It is used in Adam, RMSProp and other similar optimizers to smooth out the gradients to make the path to minimal loss a more direct and straightforward path.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-06T08:00:42.793Z
Answers.LastActivityDate: 2020-01-06T08:00:42.793Z
Answers.UserDisplayName: Clement
Body: <p>I am currently using a loss averaged over the last 100 iterations, but this leads to artifacts like the loss going down even when the current iteration has an average loss, because the loss 100 iterations ago was a large outlier.</p>
  <p>I thought about using different interval lengths, but I wonder if an average over the last few iterations really is the right way to plot the loss.</p>
  <p>Are there common alternatives? Maybe using decaying weights in the average? What are the best practices for visualizing the loss?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-04T12:01:35.153Z
Comments.Text: Why do you need to average the loss? Loss is suppose to decrease over time, so getting the average of loss over time is meaningless
Comments.UserDisplayName: Clement
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-04T12:40:50.533Z
Comments.Text: The variance of the loss per iteration is a lot larger than the decrease of the loss between the iterations. For example I currently have a loss between `2.6` and `3.2` in the last 100 iterations with an average of `2.92`. As the scatter plot is almost useless to see the trend, I visualize the average as well.
Comments.UserDisplayName: allo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-04T12:49:36.740Z
Comments.Text: Oh. Perhaps yous re looking for this: https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average
Comments.UserDisplayName: Clement
Tags: <machine-learning><objective-functions><plotting>
Title: What is the best way to smoothen out a loss curve plot?
UserDisplayName: allo
ThreadText: # What is the best way to smoothen out a loss curve plot?
  
  (Asked by: username_0 on 2020-01-03)
  
  I am currently using a loss averaged over the last 100 iterations, but this leads to artifacts like the loss going down even when the current iteration has an average loss, because the loss 100 iterations ago was a large outlier.
  
  I thought about using different interval lengths, but I wonder if an average over the last few iterations really is the right way to plot the loss.
  
  Are there common alternatives? Maybe using decaying weights in the average? What are the best practices for visualizing the loss?
  
  **Tags:** machine-learning, objective-functions, plotting
  
  > **Comments:**
  > **username_1**: Why do you need to average the loss? Loss is suppose to decrease over time, so getting the average of loss over time is meaningless  
  > **username_0**: The variance of the loss per iteration is a lot larger than the decrease of the loss between the iterations. For example I currently have a loss between `2.6` and `3.2` in the last 100 iterations with an average of `2.92`. As the scatter plot is almost useless to see the trend, I visualize the average as well.  
  > **username_1**: Oh. Perhaps yous re looking for this: https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-01-06. Score: 2
  
  You can use the [Exponential Moving Average](https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average) method. This method is used in tensorbaord as a way to smoothen a loss curve plot.
  [![enter image description here](https://i.sstatic.net/b3xGj.jpg)](https://i.sstatic.net/b3xGj.jpg)
  The algorithm is as follow:
  [![enter image description here](https://i.sstatic.net/ACJHA.jpg)](https://i.sstatic.net/ACJHA.jpg)
  However there is a small problem doing it this way. As you can see S\_t is initialized with the starting value, which makes the starting curve inaccurate.
  [![enter image description here](https://i.sstatic.net/G3V2D.png)](https://i.sstatic.net/G3V2D.png)
  The green curve is the ideal curve for the algorithm, but the purple curve is the predicted curve. The curve is not correct on the start. To solve this, a correction factor is added in, thus making the algorithm this:
  [![enter image description here](https://i.sstatic.net/uRDa6.jpg)](https://i.sstatic.net/uRDa6.jpg)
  This introduces WeightedCount which decreases over time to 0.
  
  Exponential Moving Average is also used is other areas of deep learning, the most notable being some optimization algorithms. It is used in Adam, RMSProp and other similar optimizers to smooth out the gradients to make the path to minimal loss a more direct and straightforward path.
  
  
  ---
  
  