Answers.Body: <p>So:</p>
  <ul>
  <li>&quot;<em>even a powerful neural network like XGBoost</em>&quot;: XGBoost is not a neural network</li>
  <li>&quot;<em>we can even train a simple MLP neural network with higher accuracy</em>&quot;: higher with respect to what</li>
  <li>&quot;<em>Why didn't they just look for more protein features rather than making the network more complex?</em>&quot;: because data is expensive, and maybe not even plausible to look for more... think about learning how to launch a rocket, you don't want to spend billions in trials and errors to acquire more data</li>
  </ul>
  <p>Now, onto your main question &quot;<em>So, why do highly complex neural networks like AlphaFold exist?</em>&quot;... well, just because something &quot;might&quot; work, it does not imply it &quot;will&quot; work.</p>
  <p>Indeed, a big enough 1 hidden layer MLP is a universal approximator... is this of any useful? No. It just tells you that eventually, maybe, given that you can solve an NP-hard problem, you might approximate any function.</p>
  <p>However, a decision tree or a big enough polynomial are universal approximators, and they have a very simple closed-form</p>
  <p>Now, here comes the inductive bias: yes, you can learn anything with an MLP (overfitting), but if you manage to induce some inductive bias in your model, you might hope to get a better convergence, and better generalization</p>
  <p>Now, many miss that even stacking layers, thus the reuse of share information, is already an inductive bias, thus the fact that deep networks generalize better than shallow ones</p>
  <p>Finally, you have the no-free-lunch theorem, which says roughly &quot;You cannot create a model that works the best in any circumstances... however, you can hope to create a model that is good under a certain distribution of problems</p>
  <p>Therefore, even though an MLP might work in any problem, it won't be the best one always, and thus creating highly specialized/carefully crafted models for you problem, might lead you to models that are bad for other problems, but are amazing in your class of problems</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-17T12:37:11.913Z
Answers.LastActivityDate: 2024-05-17T12:37:11.913Z
Answers.UserDisplayName: Alberto
Body: <p>If we have a sufficient number of columns (features) and a sufficient number of rows (length or volume) in a dataset that can describe a system without redundancy, we can even train a simple MLP neural network with higher accuracy.</p>
  <p>If our dataset is weak or lacks sufficient information to describe a system or data is redundant, then even a powerful neural network like <a href="https://en.wikipedia.org/wiki/XGBoost" rel="nofollow noreferrer"><em><strong>XGBoost</strong></em></a> cannot achieve anything better.</p>
  <p>So, why do highly complex neural networks like <a href="https://www.nature.com/articles/s41586-021-03819-2" rel="nofollow noreferrer"><em><strong>AlphaFold</strong></em></a> exist?</p>
  <p>Why didn't they just look for more protein features rather than making the network more complex?</p>
  <p><a href="https://i.sstatic.net/82i06y0T.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/82i06y0T.png" alt="enter image description here" /></a></p>
  
Tags: <neural-networks>
Title: Why do we need complex neural network designs?
UserDisplayName: user366312
ThreadText: # Why do we need complex neural network designs?
  
  (Asked by: username_0 on 2024-05-17)
  
  If we have a sufficient number of columns (features) and a sufficient number of rows (length or volume) in a dataset that can describe a system without redundancy, we can even train a simple MLP neural network with higher accuracy.
  
  If our dataset is weak or lacks sufficient information to describe a system or data is redundant, then even a powerful neural network like [***XGBoost***](https://en.wikipedia.org/wiki/XGBoost) cannot achieve anything better.
  
  So, why do highly complex neural networks like [***AlphaFold***](https://www.nature.com/articles/s41586-021-03819-2) exist?
  
  Why didn't they just look for more protein features rather than making the network more complex?
  
  [![enter image description here](https://i.sstatic.net/82i06y0T.png)](https://i.sstatic.net/82i06y0T.png)
  
  **Tags:** neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-17. Score: 2
  
  So:
  
  * "*even a powerful neural network like XGBoost*": XGBoost is not a neural network
  * "*we can even train a simple MLP neural network with higher accuracy*": higher with respect to what
  * "*Why didn't they just look for more protein features rather than making the network more complex?*": because data is expensive, and maybe not even plausible to look for more... think about learning how to launch a rocket, you don't want to spend billions in trials and errors to acquire more data
  
  Now, onto your main question "*So, why do highly complex neural networks like AlphaFold exist?*"... well, just because something "might" work, it does not imply it "will" work.
  
  Indeed, a big enough 1 hidden layer MLP is a universal approximator... is this of any useful? No. It just tells you that eventually, maybe, given that you can solve an NP-hard problem, you might approximate any function.
  
  However, a decision tree or a big enough polynomial are universal approximators, and they have a very simple closed-form
  
  Now, here comes the inductive bias: yes, you can learn anything with an MLP (overfitting), but if you manage to induce some inductive bias in your model, you might hope to get a better convergence, and better generalization
  
  Now, many miss that even stacking layers, thus the reuse of share information, is already an inductive bias, thus the fact that deep networks generalize better than shallow ones
  
  Finally, you have the no-free-lunch theorem, which says roughly "You cannot create a model that works the best in any circumstances... however, you can hope to create a model that is good under a certain distribution of problems
  
  Therefore, even though an MLP might work in any problem, it won't be the best one always, and thus creating highly specialized/carefully crafted models for you problem, might lead you to models that are bad for other problems, but are amazing in your class of problems
  
  
  ---
  
  