Answers.Body: <blockquote>
  <p>learning may eventually stall due to the sum of the gradients over the network inputs converging to a tiny sum even though the individual gradients for each input to the network separately stay relatively big</p>
  </blockquote>
  <p>In my opinion, this is the definition of a minima in a loss function. Infact, given any loss function
  <span class="math-container">$$
  L(\theta) = \min \sum \bar{L}(\theta)
  $$</span></p>
  <p>and it's gradient, assuming being zero:
  <span class="math-container">$$
  \nabla L(\theta) = \vec{0}
  $$</span>
  you can multiply the loss by any scalar values, and the gradient would not be impacted by it:
  <span class="math-container">$$
  \nabla L(\theta) = \nabla( 10\cdot L(\theta))
  $$</span>
  thus you can make the single gradients as big as you want, or as arbitrarily small as you want, and the cumulative gradient would still be <span class="math-container">$0$</span></p>
  <p>Just think about a logistic regression, if you just make the classes points further, the gradient will be bigger and bigger, but the global minima would stay the same</p>
  <hr />
  <p>In your case, you are assuming that theory is sound, thus assuming your neural network as enough representation power to represent your function, and that your samples are enough, that you will end up in a global minima, and that your optimizer is good enough not to be too influenced by the possible stochasticity of the minibatches... this is not usually the case, you can for sure over-parametrize your network, and it will perfectly learn to interpolate your points, but little to no guarantees on the generalization however</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-04T18:59:11.093Z
Answers.Text: Thanks for your answer! I awarded the bounty because I'm not expecting any additional answer. The thing is that indeed the minimum found is a local minimum, but it's an unwanted one, and I'm searching for standard techniques for leaving it it in the presence of symmetries in the solution space.
Answers.UserDisplayName: DCTLib
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-04T19:54:25.267Z
Answers.Text: @DCTLib unfortunately the loss landscape (convex/non convex) has little to do with  the data, but with the model you are working with, but if you find any relevant paper, please let me know, I would be interested in what they have to say
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-29T00:07:44.890Z
Answers.LastActivityDate: 2023-07-29T00:07:44.890Z
Answers.UserDisplayName: Alberto
Body: <p>What are good solution recipes to overcome the problem that in neural network learning, when the solution space has symmetries, learning may eventually stall due to the sum of the gradients over the network inputs converging to a tiny sum even though the individual gradients for each input to the network separately stay relatively big?</p>
  <p>As a simple example, consider a neural network with a two-dimensional input and a singly dimensional output with a fully connected first layer, and a MaxPool layer as the second layer. We want the network to learn the function <span class="math-container">$\max(x+2y,3x+y)$</span> for network input tuples <span class="math-container">$(x,y)$</span>. For this function, <em>two</em> different suitable weight matrices for the linear layer exist as the order of the input values to the MaxPool layer does not matter.</p>
  <p>With a standard Adam optimizer, learning sometimes stalls before the MSE is very close to <span class="math-container">$0$</span>. Small variations in the setting can cause learning to succeed, such as using a linear layer without a bias function or different random seeds. However, perhaps there are some commonly used approaches to solve such symmetry problems in a more systematic way?</p>
  <p>The following example code in PyTorch exemplifies the problem:</p>
  <pre><code>import torch
  import numpy
  from livelossplot import PlotLosses
  
  # Generate data to learn from
  numpy.random.seed(42)
  torch.manual_seed(123)
  num_samples = 1000
  X = torch.rand(num_samples, 2)
  Y = torch.max(torch.cat((X[:,0:1]+2*X[:,1:2],3*X[:,0:1]+X[:,1:2]),dim=1),dim=1,keepdim=True)[0]
  
  network = torch.nn.Sequential(
    torch.nn.Linear(X.shape[1],2,bias=True),
    torch.nn.MaxPool1d(2,1)
  )
  
  liveloss = PlotLosses()
  opt = torch.optim.Adam(network.parameters(), lr=0.02)
  lossfn = lambda x,y: ((x-y)**2).mean()
  
  for i in range(4000):
    opt.zero_grad()
    out = network(X)
    l = lossfn(Y,out)    
    current_loss = l.item()
    (l).backward()               
    opt.step()
  
    logs = {}
    logs[&quot;err&quot;] = torch.mean(torch.abs(Y - out)).item()
    liveloss.update(logs)
    liveloss.draw()
  </code></pre>
  <p>The values of the loss function over time (MSE) look as follows:</p>
  <p><a href="https://i.sstatic.net/wNMzO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wNMzO.png" alt="Loss plot of the learning process" /></a></p>
  <p>The final MSE is about 0.106. It can be verified that indeed the function to learn has a perfect representation in this network shape by initializing the weight matrix by hand:</p>
  <pre><code>with torch.no_grad():
      network[0].weight[0, 0] = 1
      network[0].weight[0, 1] = 2
      network[0].weight[1, 0] = 3
      network[0].weight[1, 1] = 1
      network[0].bias[0] = 0    
      network[0].bias[1] = 0    
  </code></pre>
  <p>When doing so before the learning process, the MSE starts with 0 and stays 0.</p>
  <p>To provide some context for why this problem is interesting, the above example is just exemplifying a problem that occurs in a different context, namely encoding domain knowledge in a neural network with a structure adapted to this knowledge shape and then using neural network learning to fine-tune the model. The domain knowledge encoding may induce many such symmetries in the solution space, and hence just wiggling some aspects of the network or the learning process in an unsystematic way is unlikely to enforce learning across all such symmetries at the same time.</p>
  <hr />
  <p><em>Edit with additional experiments based on comments.</em></p>
  <ul>
  <li>With a learning rate of 0.02 and without dropout and L2-regularization, the MSE plateaus at a MSE of 0.106 after ~10 epochs</li>
  <li>With a learning rate of 0.02 with dropout of 0.05 before the first (linear) layer and without L2-regularization, the MSE jumps between ~0.2 and ~0.25 after ~10 epochs</li>
  <li>With a learning rate of 0.02 with dropout of 0.05 after the first (linear) layer and without L2-regularization, the MSE jumps between ~0.17 and ~0.21 after ~10 epochs</li>
  </ul>
  <p>In all cases, the learned model doesn't look anywhere close to the perfect model with the linear layer weight matrix <span class="math-container">$\left(\begin{matrix}1 &amp; 2 \\ 3 &amp; 1 \end{matrix}\right)$</span> and the perfect bias vector <span class="math-container">$(0,0)^T$</span> (for the data set).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-26T02:57:00.737Z
Comments.Text: dropout comes to mind.
Comments.UserDisplayName: EngrStudent
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-26T11:25:14.263Z
Comments.Text: @EngrStudent Interesting idea. For a quick test, I've injected a "torch.nn.Dropout" layer into the model (tried probabilities: 10%, 1%, 0.1%), and it seems to make the problem even worse. Not actually surprising when thinking about it: with dropout, over node of the linear layer has to take over a bit of the functionality of the other node when it is deactivated. The hand-made "perfect" model mentioned in the case is then even a bit worse for a mean squared error function than what is learned under drop-out.
Comments.UserDisplayName: DCTLib
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-26T13:29:16.793Z
Comments.Text: Maybe your learning rate is too high, also you can try l2-regularization on the weights. Anyway, in principle you could encode symmetries in the way you represent the inputs.
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-26T14:33:58.000Z
Comments.Text: @LucaAnzalone Thanks a lot for the suggestion! In some quick experiments, L2-Regularization (giving "weight_decay=0.0001" to the Adam optimizer) didn't make a difference. Dividing the learning rate by 100 doesn't make a difference either -- the MSE plateaus at 0.106. Could you explain what you mean by encoding the symmetries in the way the inputs are represented? The function to be learned is not actually symmetric in the input value order. The only symmetry is in the solution space - there are 2 optimal weight matrices for the linear layer, which are the same except for the order of the rows.
Comments.UserDisplayName: DCTLib
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-26T16:45:51.730Z
Comments.Text: @DCTLib - can you describe the worse? Can you push the dropout in front of the layer instead of behind it? How does that modify behavior?
Comments.UserDisplayName: EngrStudent
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-26T20:34:04.307Z
Comments.Text: @EngrStudent I just added additional experimental result descriptions to the original question (at the bottom).
Comments.UserDisplayName: DCTLib
Tags: <neural-networks><training><pytorch>
Title: How to overcome symmetry in the solution space when learning a simple neural network?
UserDisplayName: DCTLib
ThreadText: # How to overcome symmetry in the solution space when learning a simple neural network?
  
  (Asked by: username_0 on 2023-07-25)
  
  What are good solution recipes to overcome the problem that in neural network learning, when the solution space has symmetries, learning may eventually stall due to the sum of the gradients over the network inputs converging to a tiny sum even though the individual gradients for each input to the network separately stay relatively big?
  As a simple example, consider a neural network with a two-dimensional input and a singly dimensional output with a fully connected first layer, and a MaxPool layer as the second layer. We want the network to learn the function $\max(x+2y,3x+y)$ for network input tuples $(x,y)$. For this function, two different suitable weight matrices for the linear layer exist as the order of the input values to the MaxPool layer does not matter.
  With a standard Adam optimizer, learning sometimes stalls before the MSE is very close to $0$. Small variations in the setting can cause learning to succeed, such as using a linear layer without a bias function or different random seeds. However, perhaps there are some commonly used approaches to solve such symmetry problems in a more systematic way?
  The following example code in PyTorch exemplifies the problem:
  import torch
  import numpy
  from livelossplot import PlotLosses
  
  # Generate data to learn from
  numpy.random.seed(42)
  torch.manual_seed(123)
  num_samples = 1000
  X = torch.rand(num_samples, 2)
  Y = torch.max(torch.cat((X[:,0:1]+2*X[:,1:2],3*X[:,0:1]+X[:,1:2]),dim=1),dim=1,keepdim=True)[0]
  
  network = torch.nn.Sequential(
    torch.nn.Linear(X.shape[1],2,bias=True),
    torch.nn.MaxPool1d(2,1)
  )
  
  liveloss = PlotLosses()
  opt = torch.optim.Adam(network.parameters(), lr=0.02)
  lossfn = lambda x,y: ((x-y)**2).mean()
  
  for i in range(4000):
    opt.zero_grad()
    out = network(X)
    l = lossfn(Y,out)    
    current_loss = l.item()
    (l).backward()               
    opt.step()
  
    logs = {}
    logs["err"] = torch.mean(torch.abs(Y - out)).item()
    liveloss.update(logs)
    liveloss.draw()
  
  The values of the loss function over time (MSE) look as follows:
  
  The final MSE is about 0.106. It can be verified that indeed the function to learn has a perfect representation in this network shape by initializing the weight matrix by hand:
  with torch.no_grad():
      network[0].weight[0, 0] = 1
      network[0].weight[0, 1] = 2
      network[0].weight[1, 0] = 3
      network[0].weight[1, 1] = 1
      network[0].bias[0] = 0    
      network[0].bias[1] = 0    
  
  When doing so before the learning process, the MSE starts with 0 and stays 0.
  To provide some context for why this problem is interesting, the above example is just exemplifying a problem that occurs in a different context, namely encoding domain knowledge in a neural network with a structure adapted to this knowledge shape and then using neural network learning to fine-tune the model. The domain knowledge encoding may induce many such symmetries in the solution space, and hence just wiggling some aspects of the network or the learning process in an unsystematic way is unlikely to enforce learning across all such symmetries at the same time.
  
  Edit with additional experiments based on comments.
  
  With a learning rate of 0.02 and without dropout and L2-regularization, the MSE plateaus at a MSE of 0.106 after ~10 epochs
  With a learning rate of 0.02 with dropout of 0.05 before the first (linear) layer and without L2-regularization, the MSE jumps between ~0.2 and ~0.25 after ~10 epochs
  With a learning rate of 0.02 with dropout of 0.05 after the first (linear) layer and without L2-regularization, the MSE jumps between ~0.17 and ~0.21 after ~10 epochs
  
  In all cases, the learned model doesn't look anywhere close to the perfect model with the linear layer weight matrix $\left(\begin{matrix}1 & 2 \\ 3 & 1 \end{matrix}\right)$ and the perfect bias vector $(0,0)^T$ (for the data set).
  
  **Tags:** neural-networks, training, pytorch
  > **Comments:**
  > **username_1**: dropout comes to mind.  
  > **username_0**: @EngrStudent Interesting idea. For a quick test, I've injected a "torch.nn.Dropout" layer into the model (tried probabilities: 10%, 1%, 0.1%), and it seems to make the problem even worse. Not actually surprising when thinking about it: with dropout, over node of the linear layer has to take over a bit of the functionality of the other node when it is deactivated. The hand-made "perfect" model mentioned in the case is then even a bit worse for a mean squared error function than what is learned under drop-out.  
  > **username_2**: Maybe your learning rate is too high, also you can try l2-regularization on the weights. Anyway, in principle you could encode symmetries in the way you represent the inputs.  
  > **username_0**: @LucaAnzalone Thanks a lot for the suggestion! In some quick experiments, L2-Regularization (giving "weight_decay=0.0001" to the Adam optimizer) didn't make a difference. Dividing the learning rate by 100 doesn't make a difference either -- the MSE plateaus at 0.106. Could you explain what you mean by encoding the symmetries in the way the inputs are represented? The function to be learned is not actually symmetric in the input value order. The only symmetry is in the solution space - there are 2 optimal weight matrices for the linear layer, which are the same except for the order of the rows.  
  > **username_1**: @DCTLib - can you describe the worse? Can you push the dropout in front of the layer instead of behind it? How does that modify behavior?  
  > **username_0**: @EngrStudent I just added additional experimental result descriptions to the original question (at the bottom).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-07-29. Score: 1
  
  > learning may eventually stall due to the sum of the gradients over the network inputs converging to a tiny sum even though the individual gradients for each input to the network separately stay relatively big
  
  In my opinion, this is the definition of a minima in a loss function. Infact, given any loss function
  $$
  L(\theta) = \min \sum \bar{L}(\theta)
  $$
  
  and it's gradient, assuming being zero:
  $$
  \nabla L(\theta) = \vec{0}
  $$
  you can multiply the loss by any scalar values, and the gradient would not be impacted by it:
  $$
  \nabla L(\theta) = \nabla( 10\cdot L(\theta))
  $$
  thus you can make the single gradients as big as you want, or as arbitrarily small as you want, and the cumulative gradient would still be $0$
  
  Just think about a logistic regression, if you just make the classes points further, the gradient will be bigger and bigger, but the global minima would stay the same
  
  ---
  
  In your case, you are assuming that theory is sound, thus assuming your neural network as enough representation power to represent your function, and that your samples are enough, that you will end up in a global minima, and that your optimizer is good enough not to be too influenced by the possible stochasticity of the minibatches... this is not usually the case, you can for sure over-parametrize your network, and it will perfectly learn to interpolate your points, but little to no guarantees on the generalization however
  
  > **Comments:**
  > **username_0**: Thanks for your answer! I awarded the bounty because I'm not expecting any additional answer. The thing is that indeed the minimum found is a local minimum, but it's an unwanted one, and I'm searching for standard techniques for leaving it it in the presence of symmetries in the solution space.  
  > **username_3**: @DCTLib unfortunately the loss landscape (convex/non convex) has little to do with  the data, but with the model you are working with, but if you find any relevant paper, please let me know, I would be interested in what they have to say  
  
  ---
  
  