Answers.Body: <p>I know that when I teach students about fully connected layers, I always introduce the topic with linear regression.  There are a few reasons:</p>
  <ul>
  <li><p>The general function applied in each dense neuron is <span class="math-container">$\sigma(wx+b)$</span>. While the nonlinearity introduced by whichever activation is chosen for <span class="math-container">$\sigma$</span> makes this nonlinear, the heart of the learning is the parameters <span class="math-container">$w$</span> and <span class="math-container">$b$</span>, or <span class="math-container">$\theta_0$</span> and <span class="math-container">$\theta_1$</span>. This provides an excellent, simplified example to work through the gradient calculation using something like <span class="math-container">$MSE$</span>.</p>
  </li>
  <li><p>The use of the linear regression also provides an excellent opportunity to reinforce the Matrix Composition Theorem. I.e., any series of linear or affine transformations <em>must</em> collapse to a single linear transformation matrix. This helps to develop the intuition for why a nonlinear activation is necessary.</p>
  </li>
  </ul>
  <p>With these main points introduced and understood, it's easier for learners to grasp the &quot;why&quot; of the design and the functioning of a dense layer without needing to rigorously work through an example by hand.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-22T18:56:56.127Z
Answers.Text: by your reasoning, there fact that a non linearity is there, is merely to avoid going back to N-successive-linear transformations. However, polynomial activations break the MC Theorem, yet MLP with polynomial activations are not universal approximators.
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-22T21:11:26.960Z
Answers.Text: This is true.. but not the point of the post or the mention of activations. The question was asking why introduce linear regression to lead to neural networks, not why do we use the specific activations that we use.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-21T12:15:21.073Z
Answers.LastActivityDate: 2024-12-21T12:15:21.073Z
Answers.UserDisplayName: David Hoelzer
Answers.Body: <p>It's just evidence that sometimes parallelisms are taken too far. NN and LR don't really share much...</p>
  <hr />
  <p>The only grounded parallelism between NN and LR is interpreting the first N+1 layers of the NN as a learned preprocessing of the input.</p>
  <p>The fact that LR has a closed form is more of a luck than a rule. You are interested in a statistic that is not the mean (such as the median)? Bye Bye closed form. Are you interested in constrained weight space? Bye Bye closed form.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-22T12:43:22.163Z
Answers.Text: I'm not sure I would call the fact that linear transformations are closed "Luck..." and it is in fact a rule or theorem that can and has been proved.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-22T18:50:35.090Z
Answers.Text: @DavidHoelzer _"and it is in fact a rule or theorem that can and has been proved."_ what? a closed form of a model is now a theorem? no. it's more luck than skills. It all boils down to the fact that expectations are closed under linear trasformations. Logistic regression is arguably as simple as linear regression, and it does not have a closed form, that's why linear regression is more a luck than a rule that we have a closed form for it
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-22T18:51:52.637Z
Answers.Text: @DavidHoelzer and i'm not saying we stubled upon it by luck, I'm saying that the fact that LinR admits a closed form, is arguably mere luck than a rule... hope I made it clear
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-22T11:39:32.093Z
Answers.LastActivityDate: 2024-12-22T11:39:32.093Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>The explicit solution, the famous <span class="math-container">$\hat\beta_{OLS} = (X^TX)^{-1}X^Ty$</span> in statistics, is a piece of linear algebra that says what the best fit is, given a certain model. There is no commentary, however, about how good that fit is. The relationship between variables and the outcome could be curvy (nonlinear); there could be interactions between the variables, that the impact one variables has on the outcome depends on the values of other variables; the interactions could be curvy (interactions between nonlinear transformations of the variables). If you just stick a straight line (hyperplane for multiple linear regression) through the data, the fit might be quite poor and able to be improved upon.</p>
  <p><a href="https://arxiv.org/abs/1806.06850" rel="nofollow noreferrer">There is an interesting paper out there that neural networks are just polynomial regressions</a>, which is a particular form of linear regression that can be solved with the matrix solution, <span class="math-container">$\hat\beta_{OLS} = (X^TX)^{-1}X^Ty$</span>. However, that requires explicit engineering of the polynomial features: which relationships are curved, which variables interact, and which interactions involve curvature. For better or for worse, neural networks do not require this. You just throw a bunch of neurons at the problem, with practitioners varying in how thoughtful they are about the exact architecture, and let the network figure out the relationship.</p>
  <p><strong>Being explicit</strong> in answering your question, neural networks offer a possible improvement in performance compared to more basic forms of linear regression (without interactions or nonlinear transformations of features), due to their high capacity for flexible modeling that the rigidity of hyperplanes do not allow. If you start allowing for complicated linear models involving interactions and curvature in the features, then linear regression could be more competitive, as <a href="https://arxiv.org/abs/1806.06850" rel="nofollow noreferrer">Cheng et al (2018)</a> argue.</p>
  <p><strong>REFERENCE</strong></p>
  <p><a href="https://arxiv.org/abs/1806.06850" rel="nofollow noreferrer">Cheng, Xi, et al. &quot;Polynomial regression as an alternative to neural nets.&quot; arXiv preprint arXiv:1806.06850 (2018).</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-23T15:59:13.563Z
Answers.LastActivityDate: 2024-12-23T15:59:13.563Z
Answers.UserDisplayName: Dave
Body: <p>I  have noticed that many introductory materials on neural networks use linear regression to predict house prices as the standard first  example on neural networks. This seems to be a common practice.</p>
  <p>But why use neural networks for a problem that already has an explicit solution, such as linear regression using the least squares criterion?</p>
  <p>Linear regression has a well-defined formula for the best-fitting line (<a href="https://en.wikipedia.org/wiki/Simple_linear_regression#Example" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Simple_linear_regression#Example</a>), so what is the benefit of applying neural networks in this case?</p>
  
Tags: <neural-networks><linear-regression><ai-field><education>
Title: Why use a neural network approach for linear regression if there is already an explicit solution
UserDisplayName: Humberto José Bortolossi
ThreadText: # Why use a neural network approach for linear regression if there is already an explicit solution
  
  (Asked by: username_0 on 2024-12-21)
  
  I have noticed that many introductory materials on neural networks use linear regression to predict house prices as the standard first example on neural networks. This seems to be a common practice.
  
  But why use neural networks for a problem that already has an explicit solution, such as linear regression using the least squares criterion?
  
  Linear regression has a well-defined formula for the best-fitting line (<https://en.wikipedia.org/wiki/Simple_linear_regression#Example>), so what is the benefit of applying neural networks in this case?
  
  **Tags:** neural-networks, linear-regression, ai-field, education
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2024-12-23. Score: 2 | Accepted answer
  
  The explicit solution, the famous $\hat\beta\_{OLS} = (X^TX)^{-1}X^Ty$ in statistics, is a piece of linear algebra that says what the best fit is, given a certain model. There is no commentary, however, about how good that fit is. The relationship between variables and the outcome could be curvy (nonlinear); there could be interactions between the variables, that the impact one variables has on the outcome depends on the values of other variables; the interactions could be curvy (interactions between nonlinear transformations of the variables). If you just stick a straight line (hyperplane for multiple linear regression) through the data, the fit might be quite poor and able to be improved upon.
  
  [There is an interesting paper out there that neural networks are just polynomial regressions](https://arxiv.org/abs/1806.06850), which is a particular form of linear regression that can be solved with the matrix solution, $\hat\beta\_{OLS} = (X^TX)^{-1}X^Ty$. However, that requires explicit engineering of the polynomial features: which relationships are curved, which variables interact, and which interactions involve curvature. For better or for worse, neural networks do not require this. You just throw a bunch of neurons at the problem, with practitioners varying in how thoughtful they are about the exact architecture, and let the network figure out the relationship.
  
  **Being explicit** in answering your question, neural networks offer a possible improvement in performance compared to more basic forms of linear regression (without interactions or nonlinear transformations of features), due to their high capacity for flexible modeling that the rigidity of hyperplanes do not allow. If you start allowing for complicated linear models involving interactions and curvature in the features, then linear regression could be more competitive, as [Cheng et al (2018)](https://arxiv.org/abs/1806.06850) argue.
  
  **REFERENCE**
  
  [Cheng, Xi, et al. "Polynomial regression as an alternative to neural nets." arXiv preprint arXiv:1806.06850 (2018).](https://arxiv.org/abs/1806.06850)
  
  
  ---
  
  ### Answer by username_1 on 2024-12-21. Score: 1
  
  I know that when I teach students about fully connected layers, I always introduce the topic with linear regression. There are a few reasons:
  
  * The general function applied in each dense neuron is $\sigma(wx+b)$. While the nonlinearity introduced by whichever activation is chosen for $\sigma$ makes this nonlinear, the heart of the learning is the parameters $w$ and $b$, or $\theta\_0$ and $\theta\_1$. This provides an excellent, simplified example to work through the gradient calculation using something like $MSE$.
  * The use of the linear regression also provides an excellent opportunity to reinforce the Matrix Composition Theorem. I.e., any series of linear or affine transformations *must* collapse to a single linear transformation matrix. This helps to develop the intuition for why a nonlinear activation is necessary.
  
  With these main points introduced and understood, it's easier for learners to grasp the "why" of the design and the functioning of a dense layer without needing to rigorously work through an example by hand.
  
  > **Comments:**
  > **username_2**: by your reasoning, there fact that a non linearity is there, is merely to avoid going back to N-successive-linear transformations. However, polynomial activations break the MC Theorem, yet MLP with polynomial activations are not universal approximators.  
  > **username_1**: This is true.. but not the point of the post or the mention of activations. The question was asking why introduce linear regression to lead to neural networks, not why do we use the specific activations that we use.  
  
  ---
  
  ### Answer by username_2 on 2024-12-22. Score: 0
  
  It's just evidence that sometimes parallelisms are taken too far. NN and LR don't really share much...
  
  ---
  
  The only grounded parallelism between NN and LR is interpreting the first N+1 layers of the NN as a learned preprocessing of the input.
  
  The fact that LR has a closed form is more of a luck than a rule. You are interested in a statistic that is not the mean (such as the median)? Bye Bye closed form. Are you interested in constrained weight space? Bye Bye closed form.
  
  > **Comments:**
  > **username_1**: I'm not sure I would call the fact that linear transformations are closed "Luck..." and it is in fact a rule or theorem that can and has been proved.  
  > **username_2**: @username_1 _"and it is in fact a rule or theorem that can and has been proved."_ what? a closed form of a model is now a theorem? no. it's more luck than skills. It all boils down to the fact that expectations are closed under linear trasformations. Logistic regression is arguably as simple as linear regression, and it does not have a closed form, that's why linear regression is more a luck than a rule that we have a closed form for it  
  > **username_2**: @username_1 and i'm not saying we stubled upon it by luck, I'm saying that the fact that LinR admits a closed form, is arguably mere luck than a rule... hope I made it clear  
  
  ---
  
  