Body: <p>I have a loss function that I'm trying to maximise using a neural network.</p>
  <p>While it does appear to increase and plateau over the training, it does so in a very &quot;noisy&quot; manner, spiking up and down, as opposed to smoothly. Below is an example (y-axis is loss, x-axis is number of epochs).</p>
  <p><a href="https://i.sstatic.net/lF9cF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lF9cF.png" alt="enter image description here" /></a></p>
  <p>I'm having a hard time diagnosing why this might be. Is it my learning rate that's too high (1e-4 with Adam optimiser)? Is it due to my neural network architecture (a MLP of structure input_dim --&gt; 512 --&gt; 256 --&gt; 128 --&gt; 1) being too complex/not complex enough? Or could it be another factor?</p>
  <p>EDIT:</p>
  <p>In response to the comment by Edoardo, I am trying to implement MINE (Mutual Information Neural Estimation). MINE computes a lower bound on the mutual information between two variables. The lower bound it computes is the supremum, over a class of functions, of a particular expression (more details can be found here <a href="https://arxiv.org/pdf/1801.04062.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1801.04062.pdf</a>). The family of functions is chosen to be a neural network of a particular architecture parametrised by parameters <span class="math-container">$\theta \in \Theta$</span> (each element in <span class="math-container">$\Theta$</span> corresponds to a unique neural network, i.e. function, in the family), and taking the supremum over  of the expression in question all possible neural networks is essentially equivalent to optimising the parameters to maximise the expression.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-22T17:03:17.647Z
Comments.Text: Why are you maximising rather than minimising the loss function? Just asking cause it's a very odd thing to do (maybe your doing reinforcement learning and dealing with a reward function rather than a loss function?). So if you have a specific reason for that it will definitely help to know it in order to answer the question.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-22T17:22:36.047Z
Comments.Text: @EdoardoGuerriero Thanks for your reply. I've added a paragraph to elaborate on why I'm maximising a "loss function". In retrospect, I guess my use of loss function was misleading - perhaps I should have said "objective function" instead.
Comments.UserDisplayName: VJ123
Tags: <neural-networks><machine-learning><feedforward-neural-networks><loss><multilayer-perceptrons>
Title: Why does my loss function fluctuate so much?
UserDisplayName: VJ123
ThreadText: # Why does my loss function fluctuate so much?
  
  (Asked by: username_0 on 2023-09-22)
  
  I have a loss function that I'm trying to maximise using a neural network.
  
  While it does appear to increase and plateau over the training, it does so in a very "noisy" manner, spiking up and down, as opposed to smoothly. Below is an example (y-axis is loss, x-axis is number of epochs).
  
  [![enter image description here](https://i.sstatic.net/lF9cF.png)](https://i.sstatic.net/lF9cF.png)
  
  I'm having a hard time diagnosing why this might be. Is it my learning rate that's too high (1e-4 with Adam optimiser)? Is it due to my neural network architecture (a MLP of structure input\_dim --> 512 --> 256 --> 128 --> 1) being too complex/not complex enough? Or could it be another factor?
  
  EDIT:
  
  In response to the comment by username_1, I am trying to implement MINE (Mutual Information Neural Estimation). MINE computes a lower bound on the mutual information between two variables. The lower bound it computes is the supremum, over a class of functions, of a particular expression (more details can be found here <https://arxiv.org/pdf/1801.04062.pdf>). The family of functions is chosen to be a neural network of a particular architecture parametrised by parameters $\theta \in \Theta$ (each element in $\Theta$ corresponds to a unique neural network, i.e. function, in the family), and taking the supremum over of the expression in question all possible neural networks is essentially equivalent to optimising the parameters to maximise the expression.
  
  **Tags:** neural-networks, machine-learning, feedforward-neural-networks, loss, multilayer-perceptrons
  
  > **Comments:**
  > **username_1**: Why are you maximising rather than minimising the loss function? Just asking cause it's a very odd thing to do (maybe your doing reinforcement learning and dealing with a reward function rather than a loss function?). So if you have a specific reason for that it will definitely help to know it in order to answer the question.  
  > **username_0**: @username_1 Thanks for your reply. I've added a paragraph to elaborate on why I'm maximising a "loss function". In retrospect, I guess my use of loss function was misleading - perhaps I should have said "objective function" instead.  
  
  ---
  
  ## 0 answers
  
  