Answers.Body: <p>You're right in the linear algebra approach here it's inaccurate to keep all the squared forms of all the 2-norms to arrive at the normal equation of the ordinary least squares (OLS) method for choosing the unknown parameters <span class="math-container">$\widehat{\theta}$</span> in a linear regression model, and bear in mind that your reference is only a draft version.</p>
  <p>Without the squares the same derivation looks ok, on one hand you get <span class="math-container">$$\|\mathbf{X\theta-y}\|_2=\|\mathbf{(X\theta-y_{\|})-y_{\perp}}\|_2 \geq \|\mathbf{y_{\perp}}\|_2$$</span>, since the same reference reason as <span class="math-container">$\mathbf{y_{\perp}}$</span> is orthogonal to both  <span class="math-container">$\mathbf{y_{\|}}$</span> and <span class="math-container">$\mathbf{X\theta}$</span> by their definition. On the other hand, by the same corresponding reference reason as triangle inequality you also get <span class="math-container">$$\|\mathbf{X\theta-y}\|_2=\|\mathbf{(X\theta-y_{\|})+(-y_{\perp})}\|_2 \leq \|\mathbf{X\theta-y_{\|}}\|_2 + \|\mathbf{-y_{\perp}}\|_2 = \|\mathbf{X\theta-y_{\|}}\|_2 + \|\mathbf{y_{\perp}}\|_2$$</span></p>
  <p>Thus finally you arrive at the same conclusion that if <span class="math-container">$\mathbf{X\theta=y_{\|}}$</span> the least squared error <span class="math-container">$\|\mathbf{X\theta-y}\|_2^2$</span> is achieved as <span class="math-container">$\|\mathbf{y_{\perp}}\|_2^2$</span> and the normal equation (3.13) must hold simultaneously.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-14T22:49:13.580Z
Answers.LastActivityDate: 2023-11-14T22:49:13.580Z
Answers.UserDisplayName: cinch
Body: <p>How  are eqs.(3.55) and (3.56) obtained? Especially, it is unclear how triangle inequality implies eq.(3.56) because we have squared norms.</p>
  <p><a href="https://i.sstatic.net/JFNUe.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JFNUe.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-12T14:07:10.077Z
Comments.Text: Please don't use screen shots for text excerpts. It makes the text inaccessible to search on this site.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-13T21:47:19.193Z
Comments.Text: Can you provide more context and details of your reference? Seems here is just plain Euclidean norm with standard Minkowski triangle inequality, there shouldn't be and no need of square in the upper right position of all your 2-norms.
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-13T22:34:00.363Z
Comments.Text: Please see page 61 of:  http://smlbook.org/book/sml-book-draft-latest.pdf
Comments.UserDisplayName: DSPinfinity
Tags: <machine-learning><linear-regression>
Title: Unclear steps in derivation of normal equations in linear regression using linear algebra approach
UserDisplayName: DSPinfinity
ThreadText: # Unclear steps in derivation of normal equations in linear regression using linear algebra approach
  
  (Asked by: username_0 on 2023-11-12)
  
  How are eqs.(3.55) and (3.56) obtained? Especially, it is unclear how triangle inequality implies eq.(3.56) because we have squared norms.
  
  [![enter image description here](https://i.sstatic.net/JFNUe.png)](https://i.sstatic.net/JFNUe.png)
  
  **Tags:** machine-learning, linear-regression
  
  > **Comments:**
  > **username_1**: Please don't use screen shots for text excerpts. It makes the text inaccessible to search on this site.  
  > **username_2**: Can you provide more context and details of your reference? Seems here is just plain Euclidean norm with standard Minkowski triangle inequality, there shouldn't be and no need of square in the upper right position of all your 2-norms.  
  > **username_0**: Please see page 61 of:  http://smlbook.org/book/sml-book-draft-latest.pdf  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-11-14. Score: 1 | Accepted answer
  
  You're right in the linear algebra approach here it's inaccurate to keep all the squared forms of all the 2-norms to arrive at the normal equation of the ordinary least squares (OLS) method for choosing the unknown parameters $\widehat{\theta}$ in a linear regression model, and bear in mind that your reference is only a draft version.
  
  Without the squares the same derivation looks ok, on one hand you get $$\|\mathbf{X\theta-y}\|\_2=\|\mathbf{(X\theta-y\_{\|})-y\_{\perp}}\|\_2 \geq \|\mathbf{y\_{\perp}}\|\_2$$, since the same reference reason as $\mathbf{y\_{\perp}}$ is orthogonal to both $\mathbf{y\_{\|}}$ and $\mathbf{X\theta}$ by their definition. On the other hand, by the same corresponding reference reason as triangle inequality you also get $$\|\mathbf{X\theta-y}\|\_2=\|\mathbf{(X\theta-y\_{\|})+(-y\_{\perp})}\|\_2 \leq \|\mathbf{X\theta-y\_{\|}}\|\_2 + \|\mathbf{-y\_{\perp}}\|\_2 = \|\mathbf{X\theta-y\_{\|}}\|\_2 + \|\mathbf{y\_{\perp}}\|\_2$$
  
  Thus finally you arrive at the same conclusion that if $\mathbf{X\theta=y\_{\|}}$ the least squared error $\|\mathbf{X\theta-y}\|\_2^2$ is achieved as $\|\mathbf{y\_{\perp}}\|\_2^2$ and the normal equation (3.13) must hold simultaneously.
  
  
  ---
  
  