Q:

Gradient of a matrix expression

What would be $\nabla f(x)$ if $f(x) = x^tAx$ where A is a n by n matrix and x is a n by 1 matrix. I have gone till $x'Ax + x^tAx'$ (where x' represents derivative) to find the solution but can see how I can write this as $(A^t + A)x$?

A:

Before we start deriving the gradient, some facts and notations for brevity:

Trace and Frobenius product relation $$\left\langle A, B C\right\rangle={\rm tr}(A^TBC) := A : B C$$ 
Cyclic properties of Trace/Frobenius product 
\begin{align}
A : B C 
 &= BC : A \\
 &= A C^T   :  B  \\
 &= {\text{etc.}} \cr
\end{align}

Let $f(x) := x^T A x = x:Ax$. 
Now, we can obtain the differential first, and then the gradient.
\begin{align}
df  
&= d\left( x:Ax \right) \\
&= \left(dx : Ax\right)  + \left(x : A \ dx\right) \\
&= \left(Ax : dx\right)  + \left(A^Tx : dx\right) \\
&= \left( A + A^T\right)x : dx\\
\end{align}
Thus, the gradient is
\begin{align}
\frac{\partial}{\partial x}f(x) =  \frac{\partial}{\partial x} \left\{x^T A x\right\} = \left( A + A^T\right)x.
\end{align}

