Answers.Body: <p>To better understand this you should think in terms of <strong>capacity</strong>. Capacity is a theoretical notion that shows how much information your network can model.</p>
  <p>The capacity of a network (given sufficient training) ties in directly with the bias/variance tradeoff:</p>
  <ul>
  <li>too little capacity and your network isn't able to learn the complex relationships in the data.</li>
  <li>too much capacity and your network has the capability of learning the noise in the dataset, besides the useful relationships.</li>
  </ul>
  <p>At some point a network reaches a point where it has a high enough capacity to <strong>memorize</strong> the whole training set!</p>
  <p>Now, by increasing the number of hidden neurons you essentially increase the capacity of your network. If the network already has enough capacity to <em>learn</em> the problem, then by increasing the neurons you are giving the network the capability of overfitting more easily.</p>
  <p><em>Note: all the above assume that the network is trained sufficiently (i.e. no early stopping, etc).</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-18T21:36:31.613Z
Answers.LastActivityDate: 2020-11-18T21:36:31.613Z
Answers.UserDisplayName: Djib2011
Answers.Body: <p>The accepted answer does not answer the question</p>
  <blockquote>
  <p>Why, if the above is correct, do we not see many neural network architectures projecting the data into higher dimensions first then reducing the size of each layer thereafter?</p>
  </blockquote>
  <p>Yes, it's true that if you increase the number of hidden neurons, you generally increase the capacity (in fact, <a href="https://ai.stackexchange.com/a/17881/2444">the VC dimension of neural networks is typically expressed as a function of the number of parameters</a>), but you're also suggesting to <em>decrease</em> the size afterwards.</p>
  <p>For example, <a href="https://www.tensorflow.org/swift/tutorials/model_training_walkthrough?hl=zh-cn" rel="nofollow noreferrer">in this tutorial</a>, they use 10 hidden neurons for the first hidden layer of an MLP, while the dataset contains only 4 features, which means that there are <span class="math-container">$4*10 = 40$</span> weights (aka parameters). In <a href="https://www.tensorflow.org/tutorials/structured_data/feature_columns" rel="nofollow noreferrer">this other tutorial</a>, there are less than 20 features (and only a few of them are used) and the first hidden layer has 128 neurons, which means that each feature is connected to <span class="math-container">$128$</span> hidden neurons, so there should be <span class="math-container">$4*128 = 512$</span> weights. So, MLPs can easily have more hidden neurons in the first layer and weights that connect the input(s) to the hidden neurons than the number of features.</p>
  <p>In the case of CNNs, you may have fewer scalar parameters because of the properties of CNNs, i.e. parameter sharing. For instance, if you have <span class="math-container">$64$</span> filters of shape <span class="math-container">$3 \times 3$</span> and you want to process grayscale images of shape <span class="math-container">$32 \times 32$</span>, then you have <span class="math-container">$3 * 3 * 64 = 576  &lt; 1024 = 32*32$</span>. Note that the number of parameters of the first layer does not change as a function of the input in the case of CNNs, so if that same CNN processes an image of size <span class="math-container">$5 \times 5$</span>, then the first layer contains more scalar parameters than pixels.</p>
  <p>So, in general, NNs can project the data to higher-dimensional space. The typical NNs that project to a lower-dimensional space are auto-encoders or data compressors, in general, that's why they are called in this way.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-24T14:53:15.797Z
Answers.LastActivityDate: 2020-11-24T15:41:34.017Z
Answers.LastEditDate: 2020-11-24T15:41:34.017Z
Answers.UserDisplayName: nbro
Body: <h2>Background</h2>
  <p>From my understanding (and following along with <a href="http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/" rel="nofollow noreferrer">this blog post</a>), (deep) neural networks apply transformations to the data such that the data's representation to the next layer (or classification layer) becomes more separate. As such, we can then apply a simple classifier(s) to the representation to chop up the regions where the different classes exist (as shown by <a href="https://medium.com/@vivek.yadav/how-neural-networks-learn-nonlinear-functions-and-classify-linearly-non-separable-data-22328e7e5be1" rel="nofollow noreferrer">this blog post</a>).</p>
  <p>If this is true and say we have some noisy data where the classes are not easily separable, would it make sense to push the input to a higher dimension, so we can more easily separate it later in the network?</p>
  <p>For example, I have some tabular data that is a bit noisy, say it has 50 dimensions (input size of 50). To me, it seems logical to project the data to a higher dimension, such that it makes it easier for the classifier to separate. In essence, I would project the data to say 60 dimensions (layer out dim = 60), so the network can represent the data with more dimensions, allowing us to linearly separate it. (I find this similar to how SVMs can classify the data by pushing it to a higher dimension).</p>
  <h2>Question</h2>
  <p>Why, if the above is correct, do we not see many neural network architectures projecting the data into higher dimensions first then reducing the size of each layer thereafter?</p>
  <p>I learned that if we have more hidden nodes than input nodes, the network will memorize rather than generalize.</p>
  
Tags: <neural-networks><overfitting><multilayer-perceptrons><generalization>
Title: Why don't neural networks project the data into higher dimensions first, then reduce the size of each layer thereafter?
UserDisplayName: Kevin
ThreadText: # Why don't neural networks project the data into higher dimensions first, then reduce the size of each layer thereafter?
  
  (Asked by: username_0 on 2020-11-18)
  
  ## Background
  
  From my understanding (and following along with [this blog post](http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/)), (deep) neural networks apply transformations to the data such that the data's representation to the next layer (or classification layer) becomes more separate. As such, we can then apply a simple classifier(s) to the representation to chop up the regions where the different classes exist (as shown by [this blog post](https://medium.com/@vivek.yadav/how-neural-networks-learn-nonlinear-functions-and-classify-linearly-non-separable-data-22328e7e5be1)).
  
  If this is true and say we have some noisy data where the classes are not easily separable, would it make sense to push the input to a higher dimension, so we can more easily separate it later in the network?
  
  For example, I have some tabular data that is a bit noisy, say it has 50 dimensions (input size of 50). To me, it seems logical to project the data to a higher dimension, such that it makes it easier for the classifier to separate. In essence, I would project the data to say 60 dimensions (layer out dim = 60), so the network can represent the data with more dimensions, allowing us to linearly separate it. (I find this similar to how SVMs can classify the data by pushing it to a higher dimension).
  
  ## Question
  
  Why, if the above is correct, do we not see many neural network architectures projecting the data into higher dimensions first then reducing the size of each layer thereafter?
  
  I learned that if we have more hidden nodes than input nodes, the network will memorize rather than generalize.
  
  **Tags:** neural-networks, overfitting, multilayer-perceptrons, generalization
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-11-18. Score: 0 | Accepted answer
  
  To better understand this you should think in terms of **capacity**. Capacity is a theoretical notion that shows how much information your network can model.
  
  The capacity of a network (given sufficient training) ties in directly with the bias/variance tradeoff:
  
  * too little capacity and your network isn't able to learn the complex relationships in the data.
  * too much capacity and your network has the capability of learning the noise in the dataset, besides the useful relationships.
  
  At some point a network reaches a point where it has a high enough capacity to **memorize** the whole training set!
  
  Now, by increasing the number of hidden neurons you essentially increase the capacity of your network. If the network already has enough capacity to *learn* the problem, then by increasing the neurons you are giving the network the capability of overfitting more easily.
  
  *Note: all the above assume that the network is trained sufficiently (i.e. no early stopping, etc).*
  
  
  ---
  
  ### Answer by username_2 on 2020-11-24. Score: 0
  
  The accepted answer does not answer the question
  
  > Why, if the above is correct, do we not see many neural network architectures projecting the data into higher dimensions first then reducing the size of each layer thereafter?
  
  Yes, it's true that if you increase the number of hidden neurons, you generally increase the capacity (in fact, [the VC dimension of neural networks is typically expressed as a function of the number of parameters](https://ai.stackexchange.com/a/17881/2444)), but you're also suggesting to *decrease* the size afterwards.
  
  For example, [in this tutorial](https://www.tensorflow.org/swift/tutorials/model_training_walkthrough?hl=zh-cn), they use 10 hidden neurons for the first hidden layer of an MLP, while the dataset contains only 4 features, which means that there are $4\*10 = 40$ weights (aka parameters). In [this other tutorial](https://www.tensorflow.org/tutorials/structured_data/feature_columns), there are less than 20 features (and only a few of them are used) and the first hidden layer has 128 neurons, which means that each feature is connected to $128$ hidden neurons, so there should be $4\*128 = 512$ weights. So, MLPs can easily have more hidden neurons in the first layer and weights that connect the input(s) to the hidden neurons than the number of features.
  
  In the case of CNNs, you may have fewer scalar parameters because of the properties of CNNs, i.e. parameter sharing. For instance, if you have $64$ filters of shape $3 \times 3$ and you want to process grayscale images of shape $32 \times 32$, then you have $3 \* 3 \* 64 = 576 < 1024 = 32\*32$. Note that the number of parameters of the first layer does not change as a function of the input in the case of CNNs, so if that same CNN processes an image of size $5 \times 5$, then the first layer contains more scalar parameters than pixels.
  
  So, in general, NNs can project the data to higher-dimensional space. The typical NNs that project to a lower-dimensional space are auto-encoders or data compressors, in general, that's why they are called in this way.
  
  
  ---
  
  