Answers.Body: <p>This is not meant as a complete answer to your question, but here are some things in your approach that arguably mess with model convergence.</p>
  <p>Firstly, optimization is all about gradients, your loss function technically <em>guides</em> the model towards outputs of <span class="math-container">$-\infty$</span> and <span class="math-container">$+\infty$</span> (prior to <span class="math-container">$\tanh$</span>), depending on the case. But what the gradient of the <span class="math-container">$\tanh(x)$</span>-function does:</p>
  <p><span class="math-container">$\lim_{x \rightarrow \infty} \tanh(x) = 1$</span> <br />
  <span class="math-container">$\lim_{x \rightarrow \infty} \tanh'(x) = 0 \quad$</span> (<span class="math-container">$\tanh'(x) = 1 - \tanh^2(x)$</span>).</p>
  <p><span class="math-container">$\lim_{x \rightarrow -\infty} \tanh(x) = -1$</span> <br />
  <span class="math-container">$\lim_{x \rightarrow -\infty} \tanh'(x) = 0$</span></p>
  <p>As soon as the model overfits to the solution where everything is <span class="math-container">$-1$</span>, there is no gradient to recover from it and this scenario is likely to happen, because <span class="math-container">$-1$</span> is the more frequent case in the training set.
  <em>&lt;-- This would be my explanation why your model converges to -1 for all inputs.</em> This is also why you statement &quot;Loss can be minimized by making Output 1&quot; is misleading. A gradient of 0 means no training regardless of the loss value.</p>
  <p>Another thing is that you are using Adam optimizer - usually a great choice but it seems not suitable in this case. Its an optimizer with momentum, which cannot deal well with rapid slope changes during training, it will basically overshoot and therefore kind of ignore the slope change - this is amplified by training on individual samples and not on batches. (A slope change happens due to your if statement - it changes from <span class="math-container">$+1$</span> to <span class="math-container">$-1$</span> or vice versa.)</p>
  <p>I'd argue its a mixture of the first two points why your special case of setting N(1) = 1 actually works: It's your very first training sample, thus it initializes the momentum of Adam which is a much better precondition not to converge to the <span class="math-container">$-1$</span> solution prematurely.</p>
  <p>If you want to keep trying with this custom function, I'd try the following 1) reduce model size 2a) Use batched training instead of individual samples to make training less noisy 2b) balance the batches such that <span class="math-container">$+1$</span> and <span class="math-container">$-1$</span> occur in equal amounts. 3) Experiment with smaller learning rates 4) You can also try other optimizers, but if everything else is configured a bit better Adam should also work. (I'd try in that order and after every change see what happens).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-12T23:05:07.450Z
Answers.Text: Doesn't the gradient of $(tanh(x) - x_0)^2$ also vanish at $\pm \infty$ though? The MSE approach works, even though the model is technically overfitting to the case where every input outputs -1 in the beginning, right? Sorry for replying so late.
Answers.UserDisplayName: Andrew Baker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-21T11:30:38.513Z
Answers.LastActivityDate: 2024-08-21T11:30:38.513Z
Answers.UserDisplayName: Chillston
Body: <p>I am working with a feedforward neural network to fit the following simple function:</p>
  <pre><code>N(1) = -1
  N(2) = -1
  N(3) = 1
  N(4) = -1
  </code></pre>
  <p>But I don't want to use the Mean-Squared Error; I'm using a custom loss function that &quot;guides&quot; the network to the correct output in a different way. My custom loss function works as follows:</p>
  <pre><code>if input == 3:
    loss = -1 * output
  else:
    loss = 1 * output
  </code></pre>
  <p>If the neural network has a <code>tanh</code> activation function, the minimizing function for this loss is clearly <code>N</code>. So the network should tend to fit <code>N</code>.</p>
  <p>But my network is only outputting <code>-1</code> (even when the input is <code>3</code>). I wrote a small sample code with pytorch to show this:</p>
  <p>Loss function:</p>
  <pre><code>import torch
  import torch.nn as nn
  
  class CustomLoss(nn.Module):
      def __init__(self):
          super().__init__()
  
      def forward(self, x, i):
  
          multiplier = -1 if i == 3 else 1
  
          return multiplier * x
  </code></pre>
  <p>Initializing Network, Optimizer, and Loss:</p>
  <pre><code>net = nn.Sequential(
      nn.Linear(1, 20),
      nn.Tanh(),
      nn.Linear(20, 20),
      nn.Tanh(),
      nn.Linear(20, 20),
      nn.Tanh(),
      nn.Linear(20, 1),
      nn.Tanh(),
  )
  optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)
  loss_function = CustomLoss()
  </code></pre>
  <p>Optimizing through Backprop:</p>
  <pre><code>for _i in range(100000):
      i = _i % 4 + 1
      input_value = torch.FloatTensor([i])
      output_value = net(input_value)
  
      optimizer.zero_grad()
      loss_value = loss_function(output_value, i)
      loss_value.backward()
      optimizer.step()
  
      print(i, output_value.item(), loss_value.item())
  </code></pre>
  <p>And after running it for a minute or so, the program outputs:</p>
  <pre><code>Input |         Output      |        Loss
  1     | -0.9999958276748657 | -0.9999958276748657
  2     | -0.9999961853027344 | -0.9999961853027344
  3     | -0.9999962449073792 |  0.9999962449073792 &lt;--- Loss can be minimized by making Output 1
  4     | -0.9999962449073792 | -0.9999962449073792
  </code></pre>
  <p>As you can see, <code>N(3)</code> is not properly fitting. So I'm led to believe the model is trapped in a local optima. But I don't know exactly why, since the MSE version of this code easily works (I tested it with the exact same setup with an altered loss function). Does anyone have any ideas?</p>
  <p>I also discovered something else: if I make <code>N(1) = 1</code> instead of <code>N(3) = 1</code> (i.e. swap the outputs of <code>1</code> and <code>3</code>), the network fits perfectly. I have absolutely no idea why this happens. It seems the loss function can approximate:</p>
  <pre><code>N(1) = 1
  N(2) = -1
  N(3) = -1
  N(4) = -1
  </code></pre>
  <p>But it can't approximate:</p>
  <pre><code>N(1) = -1
  N(2) = -1
  N(3) = 1
  N(4) = -1
  </code></pre>
  <p>Somehow.</p>
  
Tags: <python><pytorch><objective-functions><optimization>
Title: Custom Loss Function Traps Network in Local Optima
UserDisplayName: Andrew Baker
ThreadText: # Custom Loss Function Traps Network in Local Optima
  
  (Asked by: username_0 on 2024-08-15)
  
  I am working with a feedforward neural network to fit the following simple function:
  N(1) = -1
  N(2) = -1
  N(3) = 1
  N(4) = -1
  
  But I don't want to use the Mean-Squared Error; I'm using a custom loss function that "guides" the network to the correct output in a different way. My custom loss function works as follows:
  if input == 3:
    loss = -1 * output
  else:
    loss = 1 * output
  
  If the neural network has a tanh activation function, the minimizing function for this loss is clearly N. So the network should tend to fit N.
  But my network is only outputting -1 (even when the input is 3). I wrote a small sample code with pytorch to show this:
  Loss function:
  import torch
  import torch.nn as nn
  
  class CustomLoss(nn.Module):
      def __init__(self):
          super().__init__()
  
      def forward(self, x, i):
  
          multiplier = -1 if i == 3 else 1
  
          return multiplier * x
  
  Initializing Network, Optimizer, and Loss:
  net = nn.Sequential(
      nn.Linear(1, 20),
      nn.Tanh(),
      nn.Linear(20, 20),
      nn.Tanh(),
      nn.Linear(20, 20),
      nn.Tanh(),
      nn.Linear(20, 1),
      nn.Tanh(),
  )
  optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)
  loss_function = CustomLoss()
  
  Optimizing through Backprop:
  for _i in range(100000):
      i = _i % 4 + 1
      input_value = torch.FloatTensor([i])
      output_value = net(input_value)
  
      optimizer.zero_grad()
      loss_value = loss_function(output_value, i)
      loss_value.backward()
      optimizer.step()
  
      print(i, output_value.item(), loss_value.item())
  
  And after running it for a minute or so, the program outputs:
  Input |         Output      |        Loss
  1     | -0.9999958276748657 | -0.9999958276748657
  2     | -0.9999961853027344 | -0.9999961853027344
  3     | -0.9999962449073792 |  0.9999962449073792 <--- Loss can be minimized by making Output 1
  4     | -0.9999962449073792 | -0.9999962449073792
  
  As you can see, N(3) is not properly fitting. So I'm led to believe the model is trapped in a local optima. But I don't know exactly why, since the MSE version of this code easily works (I tested it with the exact same setup with an altered loss function). Does anyone have any ideas?
  I also discovered something else: if I make N(1) = 1 instead of N(3) = 1 (i.e. swap the outputs of 1 and 3), the network fits perfectly. I have absolutely no idea why this happens. It seems the loss function can approximate:
  N(1) = 1
  N(2) = -1
  N(3) = -1
  N(4) = -1
  
  But it can't approximate:
  N(1) = -1
  N(2) = -1
  N(3) = 1
  N(4) = -1
  
  Somehow.
  
  **Tags:** python, pytorch, objective-functions, optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-08-21. Score: 0
  
  This is not meant as a complete answer to your question, but here are some things in your approach that arguably mess with model convergence.
  
  Firstly, optimization is all about gradients, your loss function technically *guides* the model towards outputs of $-\infty$ and $+\infty$ (prior to $\tanh$), depending on the case. But what the gradient of the $\tanh(x)$-function does:
  
  $\lim\_{x \rightarrow \infty} \tanh(x) = 1$   
  $\lim\_{x \rightarrow \infty} \tanh'(x) = 0 \quad$ ($\tanh'(x) = 1 - \tanh^2(x)$).
  
  $\lim\_{x \rightarrow -\infty} \tanh(x) = -1$   
  $\lim\_{x \rightarrow -\infty} \tanh'(x) = 0$
  
  As soon as the model overfits to the solution where everything is $-1$, there is no gradient to recover from it and this scenario is likely to happen, because $-1$ is the more frequent case in the training set.
  *<-- This would be my explanation why your model converges to -1 for all inputs.* This is also why you statement "Loss can be minimized by making Output 1" is misleading. A gradient of 0 means no training regardless of the loss value.
  
  Another thing is that you are using Adam optimizer - usually a great choice but it seems not suitable in this case. Its an optimizer with momentum, which cannot deal well with rapid slope changes during training, it will basically overshoot and therefore kind of ignore the slope change - this is amplified by training on individual samples and not on batches. (A slope change happens due to your if statement - it changes from $+1$ to $-1$ or vice versa.)
  
  I'd argue its a mixture of the first two points why your special case of setting N(1) = 1 actually works: It's your very first training sample, thus it initializes the momentum of Adam which is a much better precondition not to converge to the $-1$ solution prematurely.
  
  If you want to keep trying with this custom function, I'd try the following 1) reduce model size 2a) Use batched training instead of individual samples to make training less noisy 2b) balance the batches such that $+1$ and $-1$ occur in equal amounts. 3) Experiment with smaller learning rates 4) You can also try other optimizers, but if everything else is configured a bit better Adam should also work. (I'd try in that order and after every change see what happens).
  
  > **Comments:**
  > **username_0**: Doesn't the gradient of $(tanh(x) - x_0)^2$ also vanish at $\pm \infty$ though? The MSE approach works, even though the model is technically overfitting to the case where every input outputs -1 in the beginning, right? Sorry for replying so late.  
  
  ---
  
  