Answers.Body: <p>One example or sample is basically a pair of data <span class="math-container">$(x_i, y_i)$</span> with <span class="math-container">$x_i$</span> randomly picked from x axis and <span class="math-container">$y_i$</span> from the piecewise function. As you can see, it doesn't provide a lot information to calculating the weighting factors. By 10,000,000 randomly samples, one may capture the turning points. But if it is not randomly sampled, I would guess the number of sample can be less. The process is different from calculating model parameters from the piecewise function parameter as the neural net doesn't know it is a piecewise function as a priori. The function it expects is a general function, which can be very complex.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-24T17:27:34.940Z
Answers.Text: Yes! My mistake, plugging that a priori information that the network must learn on its own. Nice experiment to code, checking whether uniform sampling reduces the data size needed.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-24T16:15:35.757Z
Answers.LastActivityDate: 2023-02-24T16:15:35.757Z
Answers.UserDisplayName: user115350
Body: <p>Stephen Wolfram published an interesting long <a href="https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/" rel="nofollow noreferrer">post on machine learning</a> this week.</p>
  <p>He illustrates a function approximation application with the following target function, piecewise flat with three regions.</p>
  <p><a href="https://i.sstatic.net/Xa9uW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Xa9uW.png" alt="enter image description here" /></a></p>
  <p>I understand one can describe such a function with five parameters, the three constant levels (initially low, high in the middle and mid on the right) and the two discontinuity points.</p>
  <p>As a network architecture, the following picture is given.</p>
  <p><a href="https://i.sstatic.net/0t6VT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0t6VT.png" alt="enter image description here" /></a></p>
  <p>If my count is right, there are 19 weights (4+12+3 arrows) and 8 biases (count of all neurons but the input one, 4+3+1), totalling 27 parameters. The activation function is said to be ReLU for all neurons.</p>
  <p>With this frame, we have 27 parameters in the model to estimate a 5 parameter function.</p>
  <p>The following image illustrates how the model fits the function as the number of examples grows.</p>
  <p><a href="https://i.sstatic.net/s8fZF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/s8fZF.png" alt="enter image description here" /></a></p>
  <p>From 10 thousand examples to 10 milion examples. The magnitude of data required is much higher than the complexity of the target function and the approximating network.</p>
  <blockquote>
  <p>How should this (dis)proportion of data to problem parameters be understood?</p>
  </blockquote>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-24T20:36:01.087Z
Comments.Text: Any number of factors can affect the data requirement. For example, some pairs of loss and activation functions, [such as cross-entropy logistic](https://en.m.wikipedia.org/wiki/Cross_entropy#Cross-entropy_loss_function_and_logistic_regression), are unusually good at keeping the learning rate approximately constant.
Comments.UserDisplayName: J.G.
Tags: <neural-networks>
Title: Why are neural networks so data hungry?
UserDisplayName: Jaume Oliver Lafont
ThreadText: # Why are neural networks so data hungry?
  
  (Asked by: username_0 on 2023-02-16)
  
  Stephen Wolfram published an interesting long [post on machine learning](https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/) this week.
  
  He illustrates a function approximation application with the following target function, piecewise flat with three regions.
  
  [![enter image description here](https://i.sstatic.net/Xa9uW.png)](https://i.sstatic.net/Xa9uW.png)
  
  I understand one can describe such a function with five parameters, the three constant levels (initially low, high in the middle and mid on the right) and the two discontinuity points.
  
  As a network architecture, the following picture is given.
  
  [![enter image description here](https://i.sstatic.net/0t6VT.png)](https://i.sstatic.net/0t6VT.png)
  
  If my count is right, there are 19 weights (4+12+3 arrows) and 8 biases (count of all neurons but the input one, 4+3+1), totalling 27 parameters. The activation function is said to be ReLU for all neurons.
  
  With this frame, we have 27 parameters in the model to estimate a 5 parameter function.
  
  The following image illustrates how the model fits the function as the number of examples grows.
  
  [![enter image description here](https://i.sstatic.net/s8fZF.png)](https://i.sstatic.net/s8fZF.png)
  
  From 10 thousand examples to 10 milion examples. The magnitude of data required is much higher than the complexity of the target function and the approximating network.
  
  > How should this (dis)proportion of data to problem parameters be understood?
  
  **Tags:** neural-networks
  > **Comments:**
  > **username_1**: Any number of factors can affect the data requirement. For example, some pairs of loss and activation functions, [such as cross-entropy logistic](https://en.m.wikipedia.org/wiki/Cross_entropy#Cross-entropy_loss_function_and_logistic_regression), are unusually good at keeping the learning rate approximately constant.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-02-24. Score: 2 | Accepted answer
  
  One example or sample is basically a pair of data $(x\_i, y\_i)$ with $x\_i$ randomly picked from x axis and $y\_i$ from the piecewise function. As you can see, it doesn't provide a lot information to calculating the weighting factors. By 10,000,000 randomly samples, one may capture the turning points. But if it is not randomly sampled, I would guess the number of sample can be less. The process is different from calculating model parameters from the piecewise function parameter as the neural net doesn't know it is a piecewise function as a priori. The function it expects is a general function, which can be very complex.
  
  > **Comments:**
  > **username_0**: Yes! My mistake, plugging that a priori information that the network must learn on its own. Nice experiment to code, checking whether uniform sampling reduces the data size needed.  
  
  ---
  
  