Answers.Body: <p>Note first that the first <span class="math-container">$=$</span> (equals) in <span class="math-container">$\frac{dl(\theta)}{d\theta} 
   = 0 = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$</span> should be interpreted as a "is set to", that is, we set <span class="math-container">$\frac{dl(\theta)}{d\theta} 
   = 0$</span>. Given that (apparently) <span class="math-container">$\frac{dl(\theta)}{d\theta} = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$</span>, <span class="math-container">$\frac{dl(\theta)}{d\theta} 
   = 0$</span> is equivalent to <span class="math-container">$0 = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$</span>.</p>
  
  <p>Now, let's apply some basic linear algebra:</p>
  
  <p><span class="math-container">\begin{align}
  0 &amp;= −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta) \iff \\
  0 &amp;= −(0−2X^TY + X^TX \theta) \iff \\
  0 &amp;= −0 + 2X^TY - X^TX \theta) \iff \\
  0 &amp;= 2X^TY - X^TX \theta \iff \\
  X^TX \theta &amp;= 2X^TY \iff \\
  (X^TX)^{-1}(X^TX) \theta &amp;= (X^TX)^{-1}2X^TY \iff \\
  \theta &amp;= (X^TX)^{-1}2X^TY
  \end{align}</span></p>
  
  <p>Now, you can ignore the <span class="math-container">$2$</span>, because it is just a constant, and, when optimizing, this does not influence the result.</p>
  
  <p>Note that using <span class="math-container">$\hat{\theta}$</span> instead of <span class="math-container">$\theta$</span> is just to indicate that what we will get is an "estimate" of the real <span class="math-container">$\theta$</span>, because of round off errors during the computations, etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-27T03:26:12.117Z
Answers.Text: I think it would be better to clarify under what circumstances and why, constants can be ignored. It looks at face value like doing so will halve the value of theta.
Answers.UserDisplayName: Robert Frost
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-27T10:32:40.603Z
Answers.Text: @RobertFrost I think there's a wrong derivation in the original article and that 2 would not even occur in the calculations above anymore, because it would cancel out with the 2 in the denominator. Essentially, $(Y - X\theta)^T(Y - X\theta)$ would have a term $X^TX\theta^2$ and the partial derivative of $l$ with respect to $\theta$ would produce the term $2X^TX\theta$ (but the original author of the article left out the 2 in front). In that case, after the partial derivation, all addends would have a $2$ in front, which would cancel out with the 2 at the denominator.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-27T12:01:48.243Z
Answers.Text: Maybe that's what confused the OP
Answers.UserDisplayName: Robert Frost
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-27T12:14:33.780Z
Answers.Text: @RobertFrost I think what confused the OP was that the "$= 0$" is actually "setting to $0$", but only the OP can tell us. Maybe more than one thing confused him/her.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-27T13:25:22.873Z
Answers.Text: Well , in fact, I was confused about the disappearance of 2 in the last equation. thank you very much for the detailed answer!
Answers.UserDisplayName: xava
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-26T16:25:57.063Z
Answers.LastActivityDate: 2018-11-26T16:39:15.713Z
Answers.LastEditDate: 2018-11-26T16:39:15.713Z
Answers.UserDisplayName: nbro
Body: <p>I understand both terms, linear regression and maximum likelihood, but, when it comes to the math, I am totally lost. So I am reading this article <a href="http://complx.me/2017-01-22-mle-linear-regression/" rel="nofollow noreferrer"><em>The Principle of Maximum Likelihood</em></a> (by Suriyadeepan Ramamoorthy). It is really well written, but, as mentioned in the previous sentence, I don't get the math. </p>
  
  <p>The joint probability distribution of <span class="math-container">$y,\theta, \sigma$</span> is given by (assuming <span class="math-container">$y$</span> is normally distributed): </p>
  
  <p><a href="https://i.sstatic.net/fReBa.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fReBa.gif" alt="joint probability distribution of y,θ,σ"></a></p>
  
  <p>This equivalent to maximizing the log likelihood:
  <a href="https://i.sstatic.net/Jpron.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Jpron.gif" alt="enter image description here"></a></p>
  
  <p>The maxima can be then equating through  the derivative of l(θ) to zero:
  <a href="https://i.sstatic.net/gXRPo.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gXRPo.gif" alt="enter image description here"></a></p>
  
  <p>I get everything until this point, but don't understand how this function is equivalent to the previous one :
  <a href="https://i.sstatic.net/90mST.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/90mST.gif" alt="enter image description here"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-27T12:02:57.163Z
Comments.Text: This kind of question goes well on Maths Stack Exchange.
Comments.UserDisplayName: Robert Frost
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-27T12:16:06.367Z
Comments.Text: @RobertFrost I think this type of questions can be asked here, given the topic (maximum likelihood and linear regression). AI is still a mathematical-based field, so this type of questions are normal in the AI field too. Honestly, I would like to see more questions of this kind here.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-27T15:24:53.487Z
Comments.Text: @nbro me too, I was just saying in case you wanted more.
Comments.UserDisplayName: Robert Frost
Tags: <machine-learning><linear-regression><maximum-likelihood>
Title: Understanding the math behind using maximum likelihood for linear regression
UserDisplayName: xava
ThreadText: # Understanding the math behind using maximum likelihood for linear regression
  
  (Asked by: username_0 on 2018-11-26)
  
  I understand both terms, linear regression and maximum likelihood, but, when it comes to the math, I am totally lost. So I am reading this article [*The Principle of Maximum Likelihood*](http://complx.me/2017-01-22-mle-linear-regression/) (by Suriyadeepan Ramamoorthy). It is really well written, but, as mentioned in the previous sentence, I don't get the math.
  
  The joint probability distribution of $y,\theta, \sigma$ is given by (assuming $y$ is normally distributed):
  
  [![joint probability distribution of y,θ,σ](https://i.sstatic.net/fReBa.gif)](https://i.sstatic.net/fReBa.gif)
  
  This equivalent to maximizing the log likelihood:
  [![enter image description here](https://i.sstatic.net/Jpron.gif)](https://i.sstatic.net/Jpron.gif)
  
  The maxima can be then equating through the derivative of l(θ) to zero:
  [![enter image description here](https://i.sstatic.net/gXRPo.gif)](https://i.sstatic.net/gXRPo.gif)
  
  I get everything until this point, but don't understand how this function is equivalent to the previous one :
  [![enter image description here](https://i.sstatic.net/90mST.gif)](https://i.sstatic.net/90mST.gif)
  
  **Tags:** machine-learning, linear-regression, maximum-likelihood
  > **Comments:**
  > **username_1**: This kind of question goes well on Maths Stack Exchange.  
  > **username_2**: @RobertFrost I think this type of questions can be asked here, given the topic (maximum likelihood and linear regression). AI is still a mathematical-based field, so this type of questions are normal in the AI field too. Honestly, I would like to see more questions of this kind here.  
  > **username_1**: @nbro me too, I was just saying in case you wanted more.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-11-26. Score: 2 | Accepted answer
  
  Note first that the first $=$ (equals) in $\frac{dl(\theta)}{d\theta}
  = 0 = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$ should be interpreted as a "is set to", that is, we set $\frac{dl(\theta)}{d\theta}
  = 0$. Given that (apparently) $\frac{dl(\theta)}{d\theta} = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$, $\frac{dl(\theta)}{d\theta}
  = 0$ is equivalent to $0 = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$.
  
  Now, let's apply some basic linear algebra:
  
  \begin{align}
  0 &= −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta) \iff \\
  0 &= −(0−2X^TY + X^TX \theta) \iff \\
  0 &= −0 + 2X^TY - X^TX \theta) \iff \\
  0 &= 2X^TY - X^TX \theta \iff \\
  X^TX \theta &= 2X^TY \iff \\
  (X^TX)^{-1}(X^TX) \theta &= (X^TX)^{-1}2X^TY \iff \\
  \theta &= (X^TX)^{-1}2X^TY
  \end{align}
  
  Now, you can ignore the $2$, because it is just a constant, and, when optimizing, this does not influence the result.
  
  Note that using $\hat{\theta}$ instead of $\theta$ is just to indicate that what we will get is an "estimate" of the real $\theta$, because of round off errors during the computations, etc.
  
  > **Comments:**
  > **username_1**: I think it would be better to clarify under what circumstances and why, constants can be ignored. It looks at face value like doing so will halve the value of theta.  
  > **username_2**: @RobertFrost I think there's a wrong derivation in the original article and that 2 would not even occur in the calculations above anymore, because it would cancel out with the 2 in the denominator. Essentially, $(Y - X\theta)^T(Y - X\theta)$ would have a term $X^TX\theta^2$ and the partial derivative of $l$ with respect to $\theta$ would produce the term $2X^TX\theta$ (but the original author of the article left out the 2 in front). In that case, after the partial derivation, all addends would have a $2$ in front, which would cancel out with the 2 at the denominator.  
  > **username_1**: Maybe that's what confused the OP  
  > **username_2**: @RobertFrost I think what confused the OP was that the "$= 0$" is actually "setting to $0$", but only the OP can tell us. Maybe more than one thing confused him/her.  
  > **username_0**: Well , in fact, I was confused about the disappearance of 2 in the last equation. thank you very much for the detailed answer!  
  
  ---
  
  