Answers.Body: <p>First of all, a neural network can easily be described as some high dimensional function:</p>
  <p><span class="math-container">$$A_2\sigma(A_1 \sigma(A_0 h_0 + b_0) + b_1) + b_2$$</span> where theta describes our parameters <span class="math-container">$A_0 \cdot \cdot \cdot A_k$</span> and <span class="math-container">$\sigma$</span> is some activation function like a relu or a sigmoid.</p>
  <p>For the answer to your second question, as to why we use neural networks instead of other methods, is that we don't really know why neural networks work, they just seem to work well empirically better than other methods despite the theoretical evidence. In fact if you ask top figures in this field, you'll pretty much get the same the answer</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-16T11:58:27.860Z
Answers.Text: $p(x \mid \theta) = A_2\sigma(A_1 \sigma(A_0 h_0 + b_0) + b_1) + b_2$, this is very confusing, it seems like the NN is the PDF of a density, which is definitely not
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-16T08:32:47.647Z
Answers.LastActivityDate: 2024-06-30T00:04:07.413Z
Answers.LastEditDate: 2024-06-30T00:04:07.413Z
Answers.UserDisplayName: Jayson Meribe
Body: <p>I know that neural networks are universal approximators when given a sufficient number of neurons, but there are other things that can be universal approximators, such as a Taylor series with a high enough order.</p>
  <p>So, my question is that, why can we not just optimize some high parameter count (or high dimensional) function instead? I am using a Taylor series just as an example, it can be any type of high dimensional function, and they all can be tuned with Backprop/gradient descent. I know there is lots of empirical evidence out their proving neural networks to win out over other types of functions, But I just cannot seem to understand why this is. Why does something that vaguely resembles real neurons work so well over other functions? What is the logic?</p>
  <p>PS - Maybe a dumb question, I am just a beginner that currently only sees machine learning as a calculus optimization problem :)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-15T15:45:20.010Z
Comments.Text: You're asking many questions here. What is your main question?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-15T15:48:16.617Z
Comments.Text: Main question is why prefer neural networks over any old function
Comments.UserDisplayName: Yash Nath
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-25T09:54:17.317Z
Comments.Text: You might like this arXiv paper by Chang (though Matloff seems to be the author most associated with it): [Polynomial Regression As an Alternative to Neural Nets](https://arxiv.org/abs/1806.06850).
Comments.UserDisplayName: Dave
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-27T22:51:50.810Z
Comments.Text: In theory there're many universal approximators, but in practice polynomial or Fourier series are too rigid globally while too smooth locally, you'd better need many more units than a MLP for natural problems.
Comments.UserDisplayName: cinch
Tags: <neural-networks><machine-learning><optimization><universal-approximation-theorems>
Title: Why are neural networks optimized instead of just optimizing a high dimensional function?
UserDisplayName: Yash Nath
ThreadText: # Why are neural networks optimized instead of just optimizing a high dimensional function?
  
  (Asked by: username_0 on 2024-06-15)
  
  I know that neural networks are universal approximators when given a sufficient number of neurons, but there are other things that can be universal approximators, such as a Taylor series with a high enough order.
  
  So, my question is that, why can we not just optimize some high parameter count (or high dimensional) function instead? I am using a Taylor series just as an example, it can be any type of high dimensional function, and they all can be tuned with Backprop/gradient descent. I know there is lots of empirical evidence out their proving neural networks to win out over other types of functions, But I just cannot seem to understand why this is. Why does something that vaguely resembles real neurons work so well over other functions? What is the logic?
  
  PS - Maybe a dumb question, I am just a beginner that currently only sees machine learning as a calculus optimization problem :)
  
  **Tags:** neural-networks, machine-learning, optimization, universal-approximation-theorems
  > **Comments:**
  > **username_1**: You're asking many questions here. What is your main question?  
  > **username_0**: Main question is why prefer neural networks over any old function  
  > **username_2**: You might like this arXiv paper by Chang (though Matloff seems to be the author most associated with it): [Polynomial Regression As an Alternative to Neural Nets](https://arxiv.org/abs/1806.06850).  
  > **username_3**: In theory there're many universal approximators, but in practice polynomial or Fourier series are too rigid globally while too smooth locally, you'd better need many more units than a MLP for natural problems.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2024-06-16. Score: 1
  
  First of all, a neural network can easily be described as some high dimensional function:
  
  $$A\_2\sigma(A\_1 \sigma(A\_0 h\_0 + b\_0) + b\_1) + b\_2$$ where theta describes our parameters $A\_0 \cdot \cdot \cdot A\_k$ and $\sigma$ is some activation function like a relu or a sigmoid.
  
  For the answer to your second question, as to why we use neural networks instead of other methods, is that we don't really know why neural networks work, they just seem to work well empirically better than other methods despite the theoretical evidence. In fact if you ask top figures in this field, you'll pretty much get the same the answer
  
  > **Comments:**
  > **username_5**: $p(x \mid \theta) = A_2\sigma(A_1 \sigma(A_0 h_0 + b_0) + b_1) + b_2$, this is very confusing, it seems like the NN is the PDF of a density, which is definitely not  
  
  ---
  
  