Answers.Body: <p>Gaussian noise is usually 0-mean, meaning that <em>it cancels out</em> on average especially with a squared loss (e.g. MSE): so your approach won't work.</p>
  <p>I think that you should instead <em>predict the mean and variance</em> of such noise distribution: in such a way you could handle noises that are not 0-centered too!</p>
  <p>In practice, I guess you can get inspiration from variational autoencoders. I mean, you set-up a neural net to predict the mean and log-variance. Then the loss function will be the KL divergence between the predicted Gaussian distribution (the one parametrized by mean and <code>exp(log_var)</code>) and the Gaussian prior, which corresponds to the distribution that describes the noise: <span class="math-container">$\mathcal{N}(0, 1)$</span>, in your case.</p>
  <p>In principle, with such approach you can learn also non-Gaussian noise distributions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-17T18:00:19.337Z
Answers.LastActivityDate: 2023-05-17T18:00:19.337Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am trying to predict noise (random gaussian) with the help of a neural network. I am implementing a L2 loss (torch.nn.function.mse_loss) for computing the loss function between the prediction distribution and input distribution (random gaussian). Any suggestions on what I may do to correct this?</p>
  <p><strong>1. For epoch 1 : Train loss 1.0933226346969604</strong></p>
  <p><a href="https://i.sstatic.net/QRQFT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QRQFT.png" alt="enter image description here" /></a></p>
  <p><strong>2. For epoch 200 : Train loss 1.000555157661438</strong></p>
  <p><a href="https://i.sstatic.net/nCm9g.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nCm9g.png" alt="enter image description here" /></a></p>
  <p><strong>3. For epoch 400 : Train loss: 1.174836277961731</strong></p>
  <p><a href="https://i.sstatic.net/G1ssU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/G1ssU.png" alt="enter image description here" /></a></p>
  
Tags: <neural-networks><pytorch><mean-squared-error>
Title: Loss function not able to capture the maxima of probability distribution
UserDisplayName: Formal_this
ThreadText: # Loss function not able to capture the maxima of probability distribution
  
  (Asked by: username_0 on 2023-05-17)
  
  I am trying to predict noise (random gaussian) with the help of a neural network. I am implementing a L2 loss (torch.nn.function.mse\_loss) for computing the loss function between the prediction distribution and input distribution (random gaussian). Any suggestions on what I may do to correct this?
  
  **1. For epoch 1 : Train loss 1.0933226346969604**
  
  [![enter image description here](https://i.sstatic.net/QRQFT.png)](https://i.sstatic.net/QRQFT.png)
  
  **2. For epoch 200 : Train loss 1.000555157661438**
  
  [![enter image description here](https://i.sstatic.net/nCm9g.png)](https://i.sstatic.net/nCm9g.png)
  
  **3. For epoch 400 : Train loss: 1.174836277961731**
  
  [![enter image description here](https://i.sstatic.net/G1ssU.png)](https://i.sstatic.net/G1ssU.png)
  
  **Tags:** neural-networks, pytorch, mean-squared-error
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-17. Score: 1 | Accepted answer
  
  Gaussian noise is usually 0-mean, meaning that *it cancels out* on average especially with a squared loss (e.g. MSE): so your approach won't work.
  
  I think that you should instead *predict the mean and variance* of such noise distribution: in such a way you could handle noises that are not 0-centered too!
  
  In practice, I guess you can get inspiration from variational autoencoders. I mean, you set-up a neural net to predict the mean and log-variance. Then the loss function will be the KL divergence between the predicted Gaussian distribution (the one parametrized by mean and `exp(log_var)`) and the Gaussian prior, which corresponds to the distribution that describes the noise: $\mathcal{N}(0, 1)$, in your case.
  
  In principle, with such approach you can learn also non-Gaussian noise distributions.
  
  
  ---
  
  