Answers.Body: <p>First of all, there is no real 'intelligence' innate to artificial Neural Networks (NNs).
  All they do is trying to approximate a mathematical function with a certain degree of generalization (hopefully without learning a given dataset by heart, i.e. hopefully without <em>overfitting</em>).</p>
  <p>The more nodes (or <em>neurons</em>) you include into the network, the more complex a function can be that a network can learn to approximate. It's similar to high-school math: The higher the degree of some polynomial, the better the polynomial can be adjusted to fit some observation to be modeled; with the only difference being that NNs commonly include non-linearities and are trained via some kind of stochastic gradient descent.</p>
  <p>So, yes. The more nodes a model possesses, the higher the so-called <em>model capacity</em>, i.e. the higher the degree of freedom a NN-model has to fit some function. After all, NN are said to be universal function approximators - given they have enough internal nodes in their hidden layer(s) to fit some given function.</p>
  <p>In practice, however, you don't want to blow up a model architecture unnecessarily, since this commonly results in overfitting if it doesn't cause some instabilities of the training procedure instead.</p>
  <p>Generally, the larger the model to be trained, the higher the computational cost to train the network.</p>
  <p>A common suggestion is to reduce the number of nodes in a network at the expense of increasing a network's depth, i.e. the number of hidden layers. Often, that can help reduce the demand for excessively many nodes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-13T00:23:05.657Z
Answers.LastActivityDate: 2020-07-13T00:23:05.657Z
Answers.UserDisplayName: Daniel B.
Body: <p>Just came across <a href="https://www.gwern.net/newsletter/2020/05#gpt-3" rel="nofollow noreferrer">this article on GPT-3</a>, and that lead me to the question:</p>
  <p>In order to make a certain kind of neural network architecture smarter all one needs to do is to make it bigger?</p>
  <p>Also, if that is true, how does the importance of computer power relates with the importance of fine-tuning/algorithmic improvement?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-12T12:57:37.633Z
Comments.Text: Maybe you should summarise what the article says (for those not interested in reading it) that made you think that a bigger architecture leads to more "intelligence". The thing that increases if the number of parameters of the model increases is the _capacity_ (i.e. roughly, the number of functions that the neural network can represent). So, what's the definition of intelligence you are using here?
Comments.UserDisplayName: nbro
Tags: <neural-networks><algorithm><architecture><gpt>
Title: Is the size of a neural network directly linked with an increase in its inteligence?
UserDisplayName: Gonçalo Peres
ThreadText: # Is the size of a neural network directly linked with an increase in its inteligence?
  
  (Asked by: username_0 on 2020-07-12)
  
  Just came across [this article on GPT-3](https://www.gwern.net/newsletter/2020/05#gpt-3), and that lead me to the question:
  
  In order to make a certain kind of neural network architecture smarter all one needs to do is to make it bigger?
  
  Also, if that is true, how does the importance of computer power relates with the importance of fine-tuning/algorithmic improvement?
  
  **Tags:** neural-networks, algorithm, architecture, gpt
  > **Comments:**
  > **username_1**: Maybe you should summarise what the article says (for those not interested in reading it) that made you think that a bigger architecture leads to more "intelligence". The thing that increases if the number of parameters of the model increases is the _capacity_ (i.e. roughly, the number of functions that the neural network can represent). So, what's the definition of intelligence you are using here?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-13. Score: 2 | Accepted answer
  
  First of all, there is no real 'intelligence' innate to artificial Neural Networks (NNs).
  All they do is trying to approximate a mathematical function with a certain degree of generalization (hopefully without learning a given dataset by heart, i.e. hopefully without *overfitting*).
  
  The more nodes (or *neurons*) you include into the network, the more complex a function can be that a network can learn to approximate. It's similar to high-school math: The higher the degree of some polynomial, the better the polynomial can be adjusted to fit some observation to be modeled; with the only difference being that NNs commonly include non-linearities and are trained via some kind of stochastic gradient descent.
  
  So, yes. The more nodes a model possesses, the higher the so-called *model capacity*, i.e. the higher the degree of freedom a NN-model has to fit some function. After all, NN are said to be universal function approximators - given they have enough internal nodes in their hidden layer(s) to fit some given function.
  
  In practice, however, you don't want to blow up a model architecture unnecessarily, since this commonly results in overfitting if it doesn't cause some instabilities of the training procedure instead.
  
  Generally, the larger the model to be trained, the higher the computational cost to train the network.
  
  A common suggestion is to reduce the number of nodes in a network at the expense of increasing a network's depth, i.e. the number of hidden layers. Often, that can help reduce the demand for excessively many nodes.
  
  
  ---
  
  