<p>Let $\mathbf A,\mathbf B$ be $m\times n$ matrices (or vectors is either $m$ or $n$ is $1$), 
\begin{align}
(\mathbf A+\mathbf B)^T&amp;=\mathbf A^T+\mathbf B^T \tag{1}\\
(\mathbf A\mathbf B)^T&amp;=\mathbf B^T\mathbf A^T \tag{2}
\end{align}</p>

<p>Thus, 
\begin{align}
(\mathbf y-\mathbf X\beta)^T(\mathbf y-\mathbf X\beta)&amp;=(\mathbf y^T-(\mathbf X\beta)^T)(\mathbf y-\mathbf X\beta)\\
&amp;=(\mathbf y^T-\beta^T\mathbf X^T)(\mathbf y-\mathbf X\beta)\\
&amp;=\mathbf y^T(\mathbf y-\mathbf X\beta)-\beta^T\mathbf X^T(\mathbf y-\mathbf X\beta)\\
&amp;=\mathbf y^T\mathbf y-\mathbf y^T\mathbf X\beta-\beta^T\mathbf X^T\mathbf y+\beta^T\mathbf X^T\mathbf X\beta \\
&amp;=\mathbf y^T\mathbf y-2\mathbf y^T\mathbf X\beta+\beta^T\mathbf X^T\mathbf X\beta \tag{3}
\end{align}
where line 1 uses $(1)$, line 2 uses $(2)$, and the last line follows from the fact that $\mathbf y^T\mathbf X\beta$ and $\beta^T\mathbf X^T\mathbf y$ are both $1\times1$ scalars. </p>

<hr>

<p>Let $\mathbf a,\mathbf x$ be $n\times 1$ vectors and $\mathbf A$ a symmetric $n\times n$ matrix. Matrix calculus has the following rules: 
\begin{align}
\frac{\mathrm d\ \mathbf a^T\mathbf z}{\mathrm d\ \mathbf z}&amp;=\mathbf a \tag{4} \\
\frac{\mathrm d\ \mathbf z^T\mathbf A\mathbf z}{\mathrm d\ \mathbf z}&amp;=2\mathbf A\mathbf z \tag{5}
\end{align}</p>

<p>Applying these to differentiating $(3)$ with respect to $\mathbf\beta$, we get the first order condition:
\begin{equation}
-2\mathbf X^T\mathbf y+2\mathbf X^T\mathbf X\beta=0
\end{equation}
Solving for $\beta$, we get the usual OLS formula.</p>

CC BY-SA 4.0
2018-08-03T04:07:07.397Z
Thanks. Can you recommend any source to see the proof of rules that you mentioned for the differentiation?
shk910
CC BY-SA 4.0
2018-08-03T04:28:55.087Z
@SihyunKim the Wikipedia entry on matrix calculus is a good start.
Herr K.
CC BY-SA 4.0
2018-08-03T03:38:48.557Z
2018-08-03T03:38:48.557Z
Herr K.
<blockquote>
  <ol>
  <li>Write the sum of residuals in matrix form. </li>
  </ol>
</blockquote>

<p>Attempt: $(y-X\boldsymbol{\beta})^T(y-X\boldsymbol{\beta}) = y^Ty-\boldsymbol{\beta}^TX^Ty-y^TX\boldsymbol{\beta}+\boldsymbol{\beta}^TX^TX\boldsymbol{\beta} = y^Ty-2y^TX\boldsymbol{\beta}+\boldsymbol{\beta}^TX^TX\boldsymbol{\beta}.$</p>

<blockquote>
  <ol start="2">
  <li>Minimize this and verify that $\hat{\boldsymbol{\beta}} = (X^TX)^{-1}X^Ty$. </li>
  </ol>
</blockquote>

<p>Attempt: $\frac{\partial SSR(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -2y^TX+2X^TX\boldsymbol{\beta} = 0 \rightarrow X^TX\boldsymbol{\beta}=y^TX \rightarrow \hat{\boldsymbol{\beta}} = (X^TX)^{-1}y^TX = (X^TX)^{-1}X^Ty.$</p>

<p>I haven't studied matrix calculus, so I am a little bit confused about the step in calculation. In the first problem, could you show me which rule is used for the first equality in the equation. And in the second problem, I am a little confused about the first step of differentiation. </p>

<p>I really appreciate if you show the calculation step by step. </p>

<econometrics>
Sum of residuals and Matrix calculus (step by step)
shk910
# Sum of residuals and Matrix calculus (step by step)

(Asked by: username_0 on 2018-08-03)

> 1. Write the sum of residuals in matrix form.

Attempt: $(y-X\boldsymbol{\beta})^T(y-X\boldsymbol{\beta}) = y^Ty-\boldsymbol{\beta}^TX^Ty-y^TX\boldsymbol{\beta}+\boldsymbol{\beta}^TX^TX\boldsymbol{\beta} = y^Ty-2y^TX\boldsymbol{\beta}+\boldsymbol{\beta}^TX^TX\boldsymbol{\beta}.$

> 2. Minimize this and verify that $\hat{\boldsymbol{\beta}} = (X^TX)^{-1}X^Ty$.

Attempt: $\frac{\partial SSR(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -2y^TX+2X^TX\boldsymbol{\beta} = 0 \rightarrow X^TX\boldsymbol{\beta}=y^TX \rightarrow \hat{\boldsymbol{\beta}} = (X^TX)^{-1}y^TX = (X^TX)^{-1}X^Ty.$

I haven't studied matrix calculus, so I am a little bit confused about the step in calculation. In the first problem, could you show me which rule is used for the first equality in the equation. And in the second problem, I am a little confused about the first step of differentiation.

I really appreciate if you show the calculation step by step.

**Tags:** econometrics


---

## 1 answers

### Answer by username_1 on 2018-08-03. Score: 5 | Accepted answer

Let $\mathbf A,\mathbf B$ be $m\times n$ matrices (or vectors is either $m$ or $n$ is $1$),
\begin{align}
(\mathbf A+\mathbf B)^T&=\mathbf A^T+\mathbf B^T \tag{1}\\
(\mathbf A\mathbf B)^T&=\mathbf B^T\mathbf A^T \tag{2}
\end{align}

Thus,
\begin{align}
(\mathbf y-\mathbf X\beta)^T(\mathbf y-\mathbf X\beta)&=(\mathbf y^T-(\mathbf X\beta)^T)(\mathbf y-\mathbf X\beta)\\
&=(\mathbf y^T-\beta^T\mathbf X^T)(\mathbf y-\mathbf X\beta)\\
&=\mathbf y^T(\mathbf y-\mathbf X\beta)-\beta^T\mathbf X^T(\mathbf y-\mathbf X\beta)\\
&=\mathbf y^T\mathbf y-\mathbf y^T\mathbf X\beta-\beta^T\mathbf X^T\mathbf y+\beta^T\mathbf X^T\mathbf X\beta \\
&=\mathbf y^T\mathbf y-2\mathbf y^T\mathbf X\beta+\beta^T\mathbf X^T\mathbf X\beta \tag{3}
\end{align}
where line 1 uses $(1)$, line 2 uses $(2)$, and the last line follows from the fact that $\mathbf y^T\mathbf X\beta$ and $\beta^T\mathbf X^T\mathbf y$ are both $1\times1$ scalars.

---

Let $\mathbf a,\mathbf x$ be $n\times 1$ vectors and $\mathbf A$ a symmetric $n\times n$ matrix. Matrix calculus has the following rules:
\begin{align}
\frac{\mathrm d\ \mathbf a^T\mathbf z}{\mathrm d\ \mathbf z}&=\mathbf a \tag{4} \\
\frac{\mathrm d\ \mathbf z^T\mathbf A\mathbf z}{\mathrm d\ \mathbf z}&=2\mathbf A\mathbf z \tag{5}
\end{align}

Applying these to differentiating $(3)$ with respect to $\mathbf\beta$, we get the first order condition:
\begin{equation}
-2\mathbf X^T\mathbf y+2\mathbf X^T\mathbf X\beta=0
\end{equation}
Solving for $\beta$, we get the usual OLS formula.

> **Comments:**
> **username_0**: Thanks. Can you recommend any source to see the proof of rules that you mentioned for the differentiation?  
> **username_1**: @SihyunKim the Wikipedia entry on matrix calculus is a good start.  

---

