Answers.Body: <p>I may have scratched the surface of a much larger problem when I asked this question. In the meantime I have read Lottery Hypothesis paper: <a href="https://arxiv.org/pdf/1803.03635.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1803.03635.pdf</a> </p>
  
  <p>Basically, if you overparameterise your network you are more likely to find a random initialisation that performs well: A winning ticket. The paper above shows that you can actually prune away the unneeded parts of the network after training. However, you need to overparameterise the network initially in order to increase the chance of randomly sampling a winning ticket configuration. </p>
  
  <p>I believe the case in my question above is a minimal example of this. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-17T16:43:01.340Z
Answers.LastActivityDate: 2020-03-17T16:43:01.340Z
Answers.UserDisplayName: Chrigi
Answers.Body: <p>You grasped a bit of the answer.</p>
  
  <blockquote>
    <p>In the latter two cases the optimisation gets stuck in suboptimal local minima. </p>
  </blockquote>
  
  <p>When you have only 2 dimensions, a local minima exists. When you have more dimensions, this minima gets harder and harder to reach, as its likelihood decreases. Intuitively, you have a lot more dimensions through which you can improve than if you only had 2 dimensions.</p>
  
  <p>The problem still exists, even with 1000 neurons you could find a specific set of weights which was a local minimum. However, it just becomes so much less likely.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-06T11:28:45.047Z
Answers.Text: But local minima will always exist. Arguably, with 4 hidden neurons there will be many more local minims than with 2, correct? So why then does it become less likely to get stuck in one?
Answers.UserDisplayName: Chrigi
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-06T12:07:53.850Z
Answers.Text: The local minima don't necessarily increase with more neurons (although they might!). Despite that, they're harder to find, because you have more dimensions, and it must be a minimum on all of those. So, a local minimum with XY just needs to be a local minimum for XY, while with 100 neurons, you'd need it to be a minimum on all 100 dimensions for backprop to settle there.
Answers.UserDisplayName: BlueMoon93
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-06T13:29:42.347Z
Answers.Text: okay if the number of local minima grows slow enough with the number of hidden neurons this makes sense. Thanks for your answer! Do you know if there are any good materials out there discussing these things. That is, what the optimisation "landscape" looks like and how it is likely to change with the complexity of the network?
Answers.UserDisplayName: Chrigi
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-06T13:46:38.770Z
Answers.Text: IIRC David Silver mentions the robustness of neural nets in [this](https://www.youtube.com/watch?v=2pWv7GOvuf0) course, but I couldn't find the precise moment. He basically describes that the net has so many parameters it makes it robust to local minima. On visualizing the landscape, it's impossible with enough inputs. You could do it with your 2 input neurons, but more than that cannot be represented visually for humans. I made a workshop and mention some displays [here](https://www.dropbox.com/s/eoaj9wfxifmgnbf/NN%20Workshop.pptx)
Answers.UserDisplayName: BlueMoon93
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-08T22:40:11.737Z
Answers.Text: Classification tasks with continuous input? Sure they can. Any image recognition problem has a continuous input, where pixels are in the range [0, 255]
Answers.UserDisplayName: BlueMoon93
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-06T10:17:42.757Z
Answers.LastActivityDate: 2018-04-06T10:17:42.757Z
Answers.UserDisplayName: BlueMoon93
Body: <p>In a nutshell: I want to understand why a one hidden layer neural network converges to a good minimum more reliably when a larger number of hidden neurons is used. Below a more detailed explanation of my experiment:</p>
  
  <p>I am working on a simple 2D XOR-like classification example to understand the effects of neural network initialization better. Here's a visualisation of the data and the desired decision boundary:
  <a href="https://i.sstatic.net/4jCnc.png" rel="noreferrer"><img src="https://i.sstatic.net/4jCnc.png" alt="enter image description here"></a></p>
  
  <p>Each blob consists of 5000 data points. The minimal complexity neural network to solve this problem is a one-hidden layer network with 2 hidden neurons. Since this architecture has the minimum number of parameters possible to solve this problem (with a NN) I would naively expect that this is also the easiest to optimise. However, this is not the case. </p>
  
  <p>I found that with random initialization this architecture converges around half of the time, where convergence depends on the signs of the weights. Specifically, I observed the following behaviour:</p>
  
  <pre><code>w1 = [[1,-1],[-1,1]], w2 = [1,1] --&gt; converges
  w1 = [[1,1],[1,1]],   w2 = [1,-1] --&gt; converges
  w1 = [[1,1],[1,1]],   w2 = [1,1] --&gt; finds only linear separation
  w1 = [[1,-1],[-1,1]], w2 = [1,-1] --&gt; finds only linear separation
  </code></pre>
  
  <p>This makes sense to me. In the latter two cases the optimisation gets stuck in suboptimal local minima. However, when increasing the number of hidden neurons to values greater than 2, the network develops a robustness to initialisation and starts to reliably converge for random values of w1 and w2. You can still find pathological examples, but with 4 hidden neurons the chance that one "path way" through the network will have non-pathological weights is larger. But happens to the rest of the network, is it just not used then? </p>
  
  <p>Does anybody understand better where this robustness comes from or perhaps can offer some literature discussing this issue?</p>
  
  <p>Some more information: this occurs in all training settings/architecture configurations I have investigated. For instance, activations=Relu, final_activation=sigmoid, Optimizer=Adam, learning_rate=0.1, cost_function=cross_entropy, biases were used in both layers. </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-05T12:49:03.407Z
Comments.Text: The number of hidden units does not only depend on the complexity of the function, but also the number of samples you have. Consult [this great reference](ftp://ftp.sas.com/pub/neural/FAQ3.html#A_hu).
Comments.UserDisplayName: BartoszKP
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-05T13:14:04.620Z
Comments.Text: @BartoszKP: Thanks a lot for the reference. It looks incredibly useful in general! In this case, I am not interested in a heuristic for choosing the optimal number of hidden units. I know the problem is solvable with 2 and overfitting/underfitting is a problem so the number of data points shouldn't be relevant. My goal is more to get an intuition into why having a network with redundant capacity seems beneficial here.
Comments.UserDisplayName: Chrigi
Tags: <neural-networks><optimization>
Title: Why does a one-layer hidden network get more robust to poor initialization with growing number of hidden neurons?
UserDisplayName: Chrigi
ThreadText: # Why does a one-layer hidden network get more robust to poor initialization with growing number of hidden neurons?
  
  (Asked by: username_0 on 2018-04-05)
  
  In a nutshell: I want to understand why a one hidden layer neural network converges to a good minimum more reliably when a larger number of hidden neurons is used. Below a more detailed explanation of my experiment:
  I am working on a simple 2D XOR-like classification example to understand the effects of neural network initialization better. Here's a visualisation of the data and the desired decision boundary:
  
  Each blob consists of 5000 data points. The minimal complexity neural network to solve this problem is a one-hidden layer network with 2 hidden neurons. Since this architecture has the minimum number of parameters possible to solve this problem (with a NN) I would naively expect that this is also the easiest to optimise. However, this is not the case. 
  I found that with random initialization this architecture converges around half of the time, where convergence depends on the signs of the weights. Specifically, I observed the following behaviour:
  w1 = [[1,-1],[-1,1]], w2 = [1,1] --> converges
  w1 = [[1,1],[1,1]],   w2 = [1,-1] --> converges
  w1 = [[1,1],[1,1]],   w2 = [1,1] --> finds only linear separation
  w1 = [[1,-1],[-1,1]], w2 = [1,-1] --> finds only linear separation
  
  This makes sense to me. In the latter two cases the optimisation gets stuck in suboptimal local minima. However, when increasing the number of hidden neurons to values greater than 2, the network develops a robustness to initialisation and starts to reliably converge for random values of w1 and w2. You can still find pathological examples, but with 4 hidden neurons the chance that one "path way" through the network will have non-pathological weights is larger. But happens to the rest of the network, is it just not used then? 
  Does anybody understand better where this robustness comes from or perhaps can offer some literature discussing this issue?
  Some more information: this occurs in all training settings/architecture configurations I have investigated. For instance, activations=Relu, final_activation=sigmoid, Optimizer=Adam, learning_rate=0.1, cost_function=cross_entropy, biases were used in both layers.
  
  **Tags:** neural-networks, optimization
  > **Comments:**
  > **username_1**: The number of hidden units does not only depend on the complexity of the function, but also the number of samples you have. Consult [this great reference](ftp://ftp.sas.com/pub/neural/FAQ3.html#A_hu).  
  > **username_0**: @BartoszKP: Thanks a lot for the reference. It looks incredibly useful in general! In this case, I am not interested in a heuristic for choosing the optimal number of hidden units. I know the problem is solvable with 2 and overfitting/underfitting is a problem so the number of data points shouldn't be relevant. My goal is more to get an intuition into why having a network with redundant capacity seems beneficial here.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2020-03-17. Score: 0 | Accepted answer
  
  I may have scratched the surface of a much larger problem when I asked this question. In the meantime I have read Lottery Hypothesis paper: <https://arxiv.org/pdf/1803.03635.pdf>
  
  Basically, if you overparameterise your network you are more likely to find a random initialisation that performs well: A winning ticket. The paper above shows that you can actually prune away the unneeded parts of the network after training. However, you need to overparameterise the network initially in order to increase the chance of randomly sampling a winning ticket configuration.
  
  I believe the case in my question above is a minimal example of this.
  
  
  ---
  
  ### Answer by username_2 on 2018-04-06. Score: 1
  
  You grasped a bit of the answer.
  
  > In the latter two cases the optimisation gets stuck in suboptimal local minima.
  
  When you have only 2 dimensions, a local minima exists. When you have more dimensions, this minima gets harder and harder to reach, as its likelihood decreases. Intuitively, you have a lot more dimensions through which you can improve than if you only had 2 dimensions.
  
  The problem still exists, even with 1000 neurons you could find a specific set of weights which was a local minimum. However, it just becomes so much less likely.
  
  > **Comments:**
  > **username_0**: But local minima will always exist. Arguably, with 4 hidden neurons there will be many more local minims than with 2, correct? So why then does it become less likely to get stuck in one?  
  > **username_2**: The local minima don't necessarily increase with more neurons (although they might!). Despite that, they're harder to find, because you have more dimensions, and it must be a minimum on all of those. So, a local minimum with XY just needs to be a local minimum for XY, while with 100 neurons, you'd need it to be a minimum on all 100 dimensions for backprop to settle there.  
  > **username_0**: okay if the number of local minima grows slow enough with the number of hidden neurons this makes sense. Thanks for your answer! Do you know if there are any good materials out there discussing these things. That is, what the optimisation "landscape" looks like and how it is likely to change with the complexity of the network?  
  > **username_2**: IIRC David Silver mentions the robustness of neural nets in [this](https://www.youtube.com/watch?v=2pWv7GOvuf0) course, but I couldn't find the precise moment. He basically describes that the net has so many parameters it makes it robust to local minima. On visualizing the landscape, it's impossible with enough inputs. You could do it with your 2 input neurons, but more than that cannot be represented visually for humans. I made a workshop and mention some displays [here](https://www.dropbox.com/s/eoaj9wfxifmgnbf/NN%20Workshop.pptx)  
  > **username_2**: Classification tasks with continuous input? Sure they can. Any image recognition problem has a continuous input, where pixels are in the range [0, 255]  
  
  ---
  
  