Answers.Body: <p>If your model is a linear one that can fit perfectly, with no noise, and no redundancy in the data, then you could in theory solve it as a system of linear equations. You can think of that as one equation with new data (not simply a linear combination of existing equations) reducing the number of free parameters - it doesn't fix any specific parameter, but reduces the number of them that you can freely choose to have any value before the remainder must be set logically to make all the equations balance correctly.</p>
  <p>If you keep the &quot;perfect fit&quot; requirement, and under some other assumptions (such as <a href="https://en.wikipedia.org/wiki/Bijection" rel="nofollow noreferrer">bijective</a> nonlinear functions) then this also applies to some non-linear models in principle, although there may be no direct way to calculate the correct values and with layering, as in neural networks, there will be multiple equivalent solutions.</p>
  <p>However, neural networks are usually trained without assumptions of noise free, non-redundant data or requiring a perfect fit. You might see that in some toy approximation examples, but those are not real use cases for neural networks (no-one really uses a neural network to calculate <code>xor</code> in any production system).</p>
  <p>There is still a loose relationship between number of trainable parameters in a neural network, and the dataset size that will help to train it. Many factors affect that though, so it is more common to explore the values experimentally, by plotting learning curves where the amount of data used to train is varied. And this is something that state-of-the-art developments do explore - for example with the latest batch of large language models like GPT, <a href="https://lifearchitect.ai/chinchilla/" rel="nofollow noreferrer">a ratio of 20 text tokens in the training data per free parameter in the model has been suggested</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-18T20:29:29.137Z
Answers.LastActivityDate: 2023-07-18T20:29:29.137Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm thinking from the math perspective of: &quot;you need as many equations as unknowns&quot;. And thinking the loss on each training example (with 1 output variable) as one such equation.</p>
  <p>Also I realize that in practice (if say you use ReLU), some neurons may be &quot;dead&quot; right off the bat &amp; that SGD isn't perfect, so it's unlikely to be &quot;exactly true&quot;. But I'm wonder if under ideal circumstances it is true or if it is approximately true in general.</p>
  <p>The intro of <a href="https://openreview.net/pdf?id=S1eK3i09YQ" rel="nofollow noreferrer">this paper</a> seems to imply that a model with N parameters fit to N training examples could be a perfect fit... But they don't directly state that...</p>
  
Tags: <deep-learning>
Title: Does 1 Training Example (Approximately) Fix 1 Parameter of a MLP network?
UserDisplayName: profPlum
ThreadText: # Does 1 Training Example (Approximately) Fix 1 Parameter of a MLP network?
  
  (Asked by: username_0 on 2023-07-18)
  
  I'm thinking from the math perspective of: "you need as many equations as unknowns". And thinking the loss on each training example (with 1 output variable) as one such equation.
  
  Also I realize that in practice (if say you use ReLU), some neurons may be "dead" right off the bat & that SGD isn't perfect, so it's unlikely to be "exactly true". But I'm wonder if under ideal circumstances it is true or if it is approximately true in general.
  
  The intro of [this paper](https://openreview.net/pdf?id=S1eK3i09YQ) seems to imply that a model with N parameters fit to N training examples could be a perfect fit... But they don't directly state that...
  
  **Tags:** deep-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-18. Score: 1
  
  If your model is a linear one that can fit perfectly, with no noise, and no redundancy in the data, then you could in theory solve it as a system of linear equations. You can think of that as one equation with new data (not simply a linear combination of existing equations) reducing the number of free parameters - it doesn't fix any specific parameter, but reduces the number of them that you can freely choose to have any value before the remainder must be set logically to make all the equations balance correctly.
  
  If you keep the "perfect fit" requirement, and under some other assumptions (such as [bijective](https://en.wikipedia.org/wiki/Bijection) nonlinear functions) then this also applies to some non-linear models in principle, although there may be no direct way to calculate the correct values and with layering, as in neural networks, there will be multiple equivalent solutions.
  
  However, neural networks are usually trained without assumptions of noise free, non-redundant data or requiring a perfect fit. You might see that in some toy approximation examples, but those are not real use cases for neural networks (no-one really uses a neural network to calculate `xor` in any production system).
  
  There is still a loose relationship between number of trainable parameters in a neural network, and the dataset size that will help to train it. Many factors affect that though, so it is more common to explore the values experimentally, by plotting learning curves where the amount of data used to train is varied. And this is something that state-of-the-art developments do explore - for example with the latest batch of large language models like GPT, [a ratio of 20 text tokens in the training data per free parameter in the model has been suggested](https://lifearchitect.ai/chinchilla/).
  
  
  ---
  
  