Answers.Body: <p>There's no point to fit a linear regression model (such as OLS) with neural network because it's really designed for non-linear models. But if you want to do that, you'll just need to set linear activation units.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-18T01:10:10.010Z
Answers.LastActivityDate: 2017-10-18T01:10:10.010Z
Answers.UserDisplayName: SmallChess
Answers.Body: <p>A neural network can be reduced to a linear regression model only if we use linear activation functions (i.e. <span class="math-container">$\sigma(x) = x$</span>), and only if we do not use any  neural network specific techniques such as convolution, residuals, etc., as shown below:</p>
  <p><span class="math-container">$\text{neural network}(x) = \sigma_n(W_{n} \sigma_{n-1}(W_{n-1}\dots\sigma_1(W_1 x + b_1) + \dots + b_{n-1}) + b_n) \\
  = W_n (W_{n-1} \dots (W_1 x + b_1) + \dots + b_{n-1}) + b_n \\
  = \left( W_n W_{n-1} \dots W_1 \right) x + \left( W_n W_{n-1} \dots W_2 \right) b_1 + \left( W_n W_{n-1} \dots W_3 \right) b_2 + \dots + W_n b_{n-1} + b_n \\
  = W_z x + b_z$</span></p>
  <p>where <span class="math-container">$W_z = \displaystyle \prod_i W_i$</span> is a weight matrix and <span class="math-container">$b_z$</span> is some vector constant.</p>
  <p>This follows the linear regression model form <span class="math-container">$y = Wx + b$</span>, where <span class="math-container">$W$</span> is the weight matrix and <span class="math-container">$b$</span> is the vector constant. As a result, you can analytically solve for <span class="math-container">$W_z$</span> and <span class="math-container">$b_z$</span> using linear regression techniques, and no longer need gradient descent.</p>
  <p>Note that for this to work well as a linear regression, you need to check for the OLS data assumptions, such as making sure the regressors have no collinearity, the residuals have no heteroskedasticity, there's no auto-regression, and that the errors are roughly normally distributed (<a href="https://www.statisticssolutions.com/assumptions-of-linear-regression/" rel="nofollow noreferrer">more info</a>). Deep neural networks with non-linear activation functions do not require these assumptions since they are universal approximators, although checking for some conditions may help make the task easier to predict (<a href="https://stats.stackexchange.com/a/472634/303597">more info*</a>).</p>
  <p>*<em>Note - this link talks specifically about time series data with neural network, but the same concept applies to any task in general.</em></p>
  <p>The neural network specific techniques, such as ReLu, convolutions, residuals, etc., is what allows the network to learn non-linear relationships, and therefore make neural networks something more than just repeated applications of linear regression.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T12:45:19.480Z
Answers.Text: Maybe the penultimate step of your demonstration is not fully clear, so it may be useful to provide a concrete example. Let's say that you have $2$ inputs $x_1$ and $x_2$ in the vector $x = [x_1, x_2]$, then you have 2 hidden layers of $2-3$ units each, and then, say, an output layer with 1 unit. You could then try to show if the NN really collapses to linear regression. Maybe you can find a simpler example that shows that, if you want.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T20:18:46.257Z
Answers.Text: @nbro I tried writing a draft with the example like $x = [x_1, x_2]$, but it became too verbose while not bringing much clarity. I just edited my post to try to make things more clear. If its still unclear, can you explain further exactly where and why?
Answers.UserDisplayName: user3667125
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T01:10:44.010Z
Answers.LastActivityDate: 2020-12-08T20:19:11.627Z
Answers.LastEditDate: 2020-12-08T20:19:11.627Z
Answers.UserDisplayName: user3667125
Answers.Body: <p>Generally speaking, you can say this:</p>
  
  <ol>
  <li><p>there <em>is</em> a relationship between neural network learning (I'm assuming a "vanilla" ANN here, no CNN's or RNN's or anything) and linear/logistic regression.</p></li>
  <li><p>But they're <em>not</em> the same thing.  Just related.  You could maybe consider them "cousins" to use a real-life analogy.</p></li>
  </ol>
  
  <p>The big obvious difference is this:  standard linear regression is, well, <strong>linear</strong>, that is, it's based on a straight line. So it can only separate points on a plane which can be separated by a straight line drawn on that plane.  An ANN however, is <strong>non-linear</strong> and can fit all sorts of crazy looking curves.  The reason why this is true has to do with a combination of the "activation" functions that are used, as well as the layering effect of your hidden layers.</p>
  
  <p>To be fair, if you extend linear regression to be polynomial regression, you can fit more complicated curves, but that has its own downsides.  And while they are also related, linear regression and polynomial regression aren't - strictly speaking - the same thing (although they may both be special cases of the same general technique).  </p>
  
  <p>All of that may be over-simplifying a bit.  If you really want a good explanation of both linear/logistic regression and ANN's and some explanation of how they relate and differ, I recommend Andrew Ng's ML courses on Coursera.  Both the original one and the new DeepLearning.ai ones.  </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T00:47:31.327Z
Answers.Text: You should probably note that NNs could just have linear activation functions, and, in that case, NNs can "collapse" to a linear regression model, though typically this is almost never done when using NNs.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-18T03:03:00.810Z
Answers.LastActivityDate: 2018-01-18T03:03:00.810Z
Answers.UserDisplayName: mindcrime
Body: <p>Is Deep Learning the repeated application of Linear Regression?</p>
  
Tags: <neural-networks><deep-learning><comparison><linear-regression><non-linear-regression>
Title: Is Deep Learning the repeated application of Linear Regression?
UserDisplayName: user8740537
ThreadText: # Is Deep Learning the repeated application of Linear Regression?
  
  (Asked by: username_0 on 2017-10-17)
  
  Is Deep Learning the repeated application of Linear Regression?
  
  **Tags:** neural-networks, deep-learning, comparison, linear-regression, non-linear-regression
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2017-10-18. Score: 3
  
  There's no point to fit a linear regression model (such as OLS) with neural network because it's really designed for non-linear models. But if you want to do that, you'll just need to set linear activation units.
  
  
  ---
  
  ### Answer by username_2 on 2020-12-08. Score: 3
  
  A neural network can be reduced to a linear regression model only if we use linear activation functions (i.e. $\sigma(x) = x$), and only if we do not use any neural network specific techniques such as convolution, residuals, etc., as shown below:
  
  $\text{neural network}(x) = \sigma\_n(W\_{n} \sigma\_{n-1}(W\_{n-1}\dots\sigma\_1(W\_1 x + b\_1) + \dots + b\_{n-1}) + b\_n) \\
  = W\_n (W\_{n-1} \dots (W\_1 x + b\_1) + \dots + b\_{n-1}) + b\_n \\
  = \left( W\_n W\_{n-1} \dots W\_1 \right) x + \left( W\_n W\_{n-1} \dots W\_2 \right) b\_1 + \left( W\_n W\_{n-1} \dots W\_3 \right) b\_2 + \dots + W\_n b\_{n-1} + b\_n \\
  = W\_z x + b\_z$
  
  where $W\_z = \displaystyle \prod\_i W\_i$ is a weight matrix and $b\_z$ is some vector constant.
  
  This follows the linear regression model form $y = Wx + b$, where $W$ is the weight matrix and $b$ is the vector constant. As a result, you can analytically solve for $W\_z$ and $b\_z$ using linear regression techniques, and no longer need gradient descent.
  
  Note that for this to work well as a linear regression, you need to check for the OLS data assumptions, such as making sure the regressors have no collinearity, the residuals have no heteroskedasticity, there's no auto-regression, and that the errors are roughly normally distributed ([more info](https://www.statisticssolutions.com/assumptions-of-linear-regression/)). Deep neural networks with non-linear activation functions do not require these assumptions since they are universal approximators, although checking for some conditions may help make the task easier to predict ([more info\*](https://stats.stackexchange.com/a/472634/303597)).
  
  \**Note - this link talks specifically about time series data with neural network, but the same concept applies to any task in general.*
  
  The neural network specific techniques, such as ReLu, convolutions, residuals, etc., is what allows the network to learn non-linear relationships, and therefore make neural networks something more than just repeated applications of linear regression.
  
  > **Comments:**
  > **username_3**: Maybe the penultimate step of your demonstration is not fully clear, so it may be useful to provide a concrete example. Let's say that you have $2$ inputs $x_1$ and $x_2$ in the vector $x = [x_1, x_2]$, then you have 2 hidden layers of $2-3$ units each, and then, say, an output layer with 1 unit. You could then try to show if the NN really collapses to linear regression. Maybe you can find a simpler example that shows that, if you want.  
  > **username_2**: @nbro I tried writing a draft with the example like $x = [x_1, x_2]$, but it became too verbose while not bringing much clarity. I just edited my post to try to make things more clear. If its still unclear, can you explain further exactly where and why?  
  
  ---
  
  ### Answer by username_4 on 2018-01-18. Score: 1
  
  Generally speaking, you can say this:
  
  1. there *is* a relationship between neural network learning (I'm assuming a "vanilla" ANN here, no CNN's or RNN's or anything) and linear/logistic regression.
  2. But they're *not* the same thing. Just related. You could maybe consider them "cousins" to use a real-life analogy.
  
  The big obvious difference is this: standard linear regression is, well, **linear**, that is, it's based on a straight line. So it can only separate points on a plane which can be separated by a straight line drawn on that plane. An ANN however, is **non-linear** and can fit all sorts of crazy looking curves. The reason why this is true has to do with a combination of the "activation" functions that are used, as well as the layering effect of your hidden layers.
  
  To be fair, if you extend linear regression to be polynomial regression, you can fit more complicated curves, but that has its own downsides. And while they are also related, linear regression and polynomial regression aren't - strictly speaking - the same thing (although they may both be special cases of the same general technique).
  
  All of that may be over-simplifying a bit. If you really want a good explanation of both linear/logistic regression and ANN's and some explanation of how they relate and differ, I recommend Andrew Ng's ML courses on Coursera. Both the original one and the new DeepLearning.ai ones.
  
  > **Comments:**
  > **username_3**: You should probably note that NNs could just have linear activation functions, and, in that case, NNs can "collapse" to a linear regression model, though typically this is almost never done when using NNs.  
  
  ---
  
  