Answers.Body: <p>Yes, it is a well known problem called <a href="https://en.wikipedia.org/wiki/Curse_of_dimensionality" rel="nofollow noreferrer">Curse of Dimensionality</a>. It happens when a finite number of data samples is used to train a network with a high-dimensional feature space (very deep network).</p>
  <p>With regard to your question: <strong>yes</strong>, smaller networks (representational spaces with lower dimensions) describe better smaller datasets.</p>
  <p>Why?</p>
  <p>Because of data sparsity. As the features to represent a single sample increases the space between samples in the network representation also increases. When the samples are represented in too sparse space (all samples are very far away from each other in this high-dimensional space) then you can not draw any conclusions or relationships about them.</p>
  <p>There is a balance between network dimensionality and dataset size, both of them must be in accordance to each other. You can think of this in this way: if data is represented in a very low dimensional space you can not tell them appart (1D space), if the data is represented in a very high dimensional space, then you can not find relationships within it. The dimensionality must be just the right one.</p>
  <p><a href="https://i.sstatic.net/945bN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/945bN.png" alt="enter image description here" /></a></p>
  <p>Image from <a href="https://deepai.org/machine-learning-glossary-and-terms/curse-of-dimensionality" rel="nofollow noreferrer">this post</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-29T07:31:54.347Z
Answers.Text: Thank you so much for your detailed explanation. :)
Answers.UserDisplayName: realmarv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-28T10:29:12.053Z
Answers.LastActivityDate: 2021-09-28T10:29:12.053Z
Answers.UserDisplayName: JVGD
Body: <p>Assume we have a neural network and we want to train it on a classification problem. The hidden layers of the neural network are kind of feature representations of the input data.</p>
  <p>If the neural network is big and the amount of data isn't enough for the complexity of the model, does it usually learn worse representations than a smaller neural network which is good for the amount of data we have?</p>
  
Tags: <neural-networks><deep-learning><overfitting><generalization><representation-learning>
Title: Does a bigger neural network learn "worse" representations than a small neural network when the amount of data isn't enough?
UserDisplayName: realmarv
ThreadText: # Does a bigger neural network learn "worse" representations than a small neural network when the amount of data isn't enough?
  
  (Asked by: username_0 on 2021-09-28)
  
  Assume we have a neural network and we want to train it on a classification problem. The hidden layers of the neural network are kind of feature representations of the input data.
  
  If the neural network is big and the amount of data isn't enough for the complexity of the model, does it usually learn worse representations than a smaller neural network which is good for the amount of data we have?
  
  **Tags:** neural-networks, deep-learning, overfitting, generalization, representation-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-28. Score: 0 | Accepted answer
  
  Yes, it is a well known problem called [Curse of Dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality). It happens when a finite number of data samples is used to train a network with a high-dimensional feature space (very deep network).
  
  With regard to your question: **yes**, smaller networks (representational spaces with lower dimensions) describe better smaller datasets.
  
  Why?
  
  Because of data sparsity. As the features to represent a single sample increases the space between samples in the network representation also increases. When the samples are represented in too sparse space (all samples are very far away from each other in this high-dimensional space) then you can not draw any conclusions or relationships about them.
  
  There is a balance between network dimensionality and dataset size, both of them must be in accordance to each other. You can think of this in this way: if data is represented in a very low dimensional space you can not tell them appart (1D space), if the data is represented in a very high dimensional space, then you can not find relationships within it. The dimensionality must be just the right one.
  
  [![enter image description here](https://i.sstatic.net/945bN.png)](https://i.sstatic.net/945bN.png)
  
  Image from [this post](https://deepai.org/machine-learning-glossary-and-terms/curse-of-dimensionality)
  
  > **Comments:**
  > **username_0**: Thank you so much for your detailed explanation. :)  
  
  ---
  
  