Answers.Body: <p>In a nutshell, if you already have a number of models, you usually should be able to distinguish (intuitively, if you will) between simpler and more complex ones. E.g. based on the number of inputs and number of layers, as you have already indicated in the question. Then, if a simpler model and a more complex model perform the same task, and the complex model does not perform significally better than the simpler one, you should use the simpler model. It's your role to decide what difference in performance would be significant, usually based on your use case. It's Occam's razor in practice (<a href="https://en.m.wikipedia.org/wiki/Occam%27s_razor" rel="nofollow noreferrer">https://en.m.wikipedia.org/wiki/Occam%27s_razor</a>).
  You might learn more practical aspects as part of this free course <a href="https://lagunita.stanford.edu/courses/HumanitiesSciences/StatLearning/Winter2016/about" rel="nofollow noreferrer">https://lagunita.stanford.edu/courses/HumanitiesSciences/StatLearning/Winter2016/about</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-26T08:06:30.010Z
Answers.LastActivityDate: 2019-01-26T08:06:30.010Z
Answers.UserDisplayName: Pawel Kranzberg
Answers.Body: <p>In what context are you asking this? It is totally different if you want to perform object detection, regression or, for example, reinforcement learning.</p>
  
  <p>For the first case I would say that main point in using simple vs complex model is size of training data. If you have 1000 training samples you can't expect large network to perform better than simple one.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-25T17:50:15.257Z
Answers.LastActivityDate: 2019-01-25T17:50:15.257Z
Answers.UserDisplayName: carobnodrvo
Answers.Body: <p>If you want to find a proper architecture for your model, you can use the <a href="https://en.wikipedia.org/wiki/Neural_architecture_search" rel="nofollow noreferrer">NAS</a> (neural architecture search) methods instead of running some naive models to find a model and involving to decide which model is more complex or simpler. Some methods which used in NAS to find a proper architecture are:</p>
  
  <ol>
  <li>NAS with Reinforcement Learning</li>
  <li>NAS with Evolution</li>
  <li>NAS with Hill-climbing</li>
  <li>Multi-objective Neural architecture search</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-14T09:54:18.070Z
Answers.Text: can you help me to understand the difference between simple vs complex model?  with the help of which parameters we can differentiate this
Answers.UserDisplayName: Maheshwar Ligade
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-14T09:42:01.280Z
Answers.LastActivityDate: 2019-01-14T09:56:13.733Z
Answers.LastEditDate: 2019-01-14T09:56:13.733Z
Answers.UserDisplayName: OmG
Answers.Body: <p>Consider a continuum of complexity in models.</p>
  
  <ul>
  <li><p><strong>Trivial:</strong> <span class="math-container">$y = x + a$</span></p></li>
  <li><p><strong>Simple:</strong> <span class="math-container">$y = x \, \log \, (a x + b) + c$</span></p></li>
  <li><p><strong>Moderately complex:</strong> A wind turbine under constant wind velocity</p></li>
  <li><p><strong>Very complex:</strong> Ray tracing of lit 3-D motion scenes to pixels</p></li>
  <li><p><strong>Astronomically complex:</strong> The weather</p></li>
  </ul>
  
  <p>Now consider a continuum regarding the generality or specificity of models.</p>
  
  <ul>
  <li><p><strong>Very specific:</strong> The robot for the Mars mission has an exact mechanical topology, materials call-out, and set of mechanical coordinates contained in the CAD files used to machine the robot's parts.</p></li>
  <li><p><strong>Somewhat specific:</strong> The formulas guiding the design of an internal combustion engine, which are well known.</p></li>
  <li><p><strong>Somewhat general:</strong> The phenomenon is deterministic and the variables and their domains are known.</p></li>
  <li><p><strong>Very general:</strong> There's probably some model because it works in nature but we know little more.</p></li>
  </ul>
  
  <p>There are twenty permutations at the above level of granularity. Every one has purpose in mathematical analysis, applied research, engineering, and monetization.</p>
  
  <p>Here are some general correlations between input, output, and layer counts.</p>
  
  <ul>
  <li><p>Higher complexity often corresponds to larger layer count.</p></li>
  <li><p>Higher i/o dimensionality corresponds to higher width to the corresponding i/o layers.</p></li>
  <li><p>Mapping generality to or from specificity generally requires complexity.</p></li>
  </ul>
  
  <p>Now, to make this answer even less appealing to those who want formula answer they can memorize, ...</p>
  
  <ul>
  <li><p>Each artificial network is a model of an arbitrary function before training and a model of a specific function afterward.</p></li>
  <li><p>Loss functions are models of disparity.</p></li>
  <li><p>An algorithm is a model of a process created by spreading a recursive definition out in time to map into a model of centralized computation called a CPU.</p></li>
  <li><p>The recursive definition is a model too.</p></li>
  </ul>
  
  <p>There is almost nothing in science that is a not a model except ideas or data that are not yet modeled.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-25T16:10:09.703Z
Answers.LastActivityDate: 2019-01-25T18:44:10.077Z
Answers.LastEditDate: 2019-01-25T18:44:10.077Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>How do you distinguish between a complex and a simple model in machine learning? Which parameters control the complexity or simplicity of a model? Is it the number of inputs, or maybe the number of layers?</p>
  <p>Moreover, when should a simple model be used instead of a complex one, and vice-versa?</p>
  
Tags: <neural-networks><machine-learning><deep-learning><models><computational-learning-theory>
Title: How do you distinguish between a complex and a simple model in machine learning?
UserDisplayName: Maheshwar Ligade
ThreadText: # How do you distinguish between a complex and a simple model in machine learning?
  
  (Asked by: username_0 on 2019-01-14)
  
  How do you distinguish between a complex and a simple model in machine learning? Which parameters control the complexity or simplicity of a model? Is it the number of inputs, or maybe the number of layers?
  
  Moreover, when should a simple model be used instead of a complex one, and vice-versa?
  
  **Tags:** neural-networks, machine-learning, deep-learning, models, computational-learning-theory
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_4 on 2019-01-25. Score: 3 | Accepted answer
  
  Consider a continuum of complexity in models.
  
  * **Trivial:** $y = x + a$
  * **Simple:** $y = x \, \log \, (a x + b) + c$
  * **Moderately complex:** A wind turbine under constant wind velocity
  * **Very complex:** Ray tracing of lit 3-D motion scenes to pixels
  * **Astronomically complex:** The weather
  
  Now consider a continuum regarding the generality or specificity of models.
  
  * **Very specific:** The robot for the Mars mission has an exact mechanical topology, materials call-out, and set of mechanical coordinates contained in the CAD files used to machine the robot's parts.
  * **Somewhat specific:** The formulas guiding the design of an internal combustion engine, which are well known.
  * **Somewhat general:** The phenomenon is deterministic and the variables and their domains are known.
  * **Very general:** There's probably some model because it works in nature but we know little more.
  
  There are twenty permutations at the above level of granularity. Every one has purpose in mathematical analysis, applied research, engineering, and monetization.
  
  Here are some general correlations between input, output, and layer counts.
  
  * Higher complexity often corresponds to larger layer count.
  * Higher i/o dimensionality corresponds to higher width to the corresponding i/o layers.
  * Mapping generality to or from specificity generally requires complexity.
  
  Now, to make this answer even less appealing to those who want formula answer they can memorize, ...
  
  * Each artificial network is a model of an arbitrary function before training and a model of a specific function afterward.
  * Loss functions are models of disparity.
  * An algorithm is a model of a process created by spreading a recursive definition out in time to map into a model of centralized computation called a CPU.
  * The recursive definition is a model too.
  
  There is almost nothing in science that is a not a model except ideas or data that are not yet modeled.
  
  
  ---
  
  ### Answer by username_1 on 2019-01-26. Score: 1
  
  In a nutshell, if you already have a number of models, you usually should be able to distinguish (intuitively, if you will) between simpler and more complex ones. E.g. based on the number of inputs and number of layers, as you have already indicated in the question. Then, if a simpler model and a more complex model perform the same task, and the complex model does not perform significally better than the simpler one, you should use the simpler model. It's your role to decide what difference in performance would be significant, usually based on your use case. It's Occam's razor in practice (<https://en.m.wikipedia.org/wiki/Occam%27s_razor>).
  You might learn more practical aspects as part of this free course <https://lagunita.stanford.edu/courses/HumanitiesSciences/StatLearning/Winter2016/about>
  
  
  ---
  
  ### Answer by username_2 on 2019-01-25. Score: 1
  
  In what context are you asking this? It is totally different if you want to perform object detection, regression or, for example, reinforcement learning.
  
  For the first case I would say that main point in using simple vs complex model is size of training data. If you have 1000 training samples you can't expect large network to perform better than simple one.
  
  
  ---
  
  ### Answer by username_3 on 2019-01-14. Score: 1
  
  If you want to find a proper architecture for your model, you can use the [NAS](https://en.wikipedia.org/wiki/Neural_architecture_search) (neural architecture search) methods instead of running some naive models to find a model and involving to decide which model is more complex or simpler. Some methods which used in NAS to find a proper architecture are:
  
  1. NAS with Reinforcement Learning
  2. NAS with Evolution
  3. NAS with Hill-climbing
  4. Multi-objective Neural architecture search
  
  > **Comments:**
  > **username_0**: can you help me to understand the difference between simple vs complex model?  with the help of which parameters we can differentiate this  
  
  ---
  
  