Question: Let $A, X\in\mathbb{R}^{n\times n}$. The scalar objective function is
$$J=\mathrm{tr}(AX)$$
If no constraints, let the derivative of $J$ with respect to $X$ be zeros, then we have
$$A=0$$
Suppose $A$ is also a complex function, from $A=0$ I can further calculate something.

**My question is**: what if $X$ is an orthogonal matrix, i.e., $X^TX=I$? Then it becomes an constrained optimization problem. Can I still use matrix differential techniques to derive the minimizer? Thanks.

Answer: 
The method of [Lagrange multipliers](http://en.wikipedia.org/wiki/Lagrange_multiplier) yields the solution. 

One studies the (real-valued) function $F(X)$ with the (matrix) constraint that $C(X)=I$ for $F(X)=\text{trace}(AX)$ and $C(X)=X^TX$, hence 
$$
F(X)=\sum\limits\_{ij}A\_{ji}X\_{ij},\quad
C\_{ij}(X)=\sum\limits\_{k}X\_{ki}X\_{kj}.
$$
The gradients of $F$ and $C$ are given by $\partial\_{ij}F(X)=A\_{ji}$ and $$
\partial\_{ij}C\_{k\ell}(X)=X\_{i\ell}[k=j]+X\_{ik}[\ell=j].
$$
One wants that there exists some *multipliers* $\lambda\_{ij}$ such that, for every $i$ and $j$, 
$$
\partial\_{ij}F(X)=\sum\limits\_{k\ell}\lambda\_{k\ell}\partial\_{ij}C\_{k\ell}(X).
$$
This condition reads
$$
A\_{ji}=\sum\limits\_{k,\ell}\lambda\_{k\ell}\left(X\_{i\ell}[k=j]+X\_{ik}[\ell=j]\right)=\sum\limits\_{\ell}\lambda\_{j\ell}X\_{i\ell}+\sum\limits\_{k}\lambda\_{kj}X\_{ik},
$$
or, equivalently, introducing the matrix $\Lambda=(\lambda\_{ij})$,
$$
A^T=X\Lambda^T+X\Lambda.
$$
The matrix $M=\Lambda^T+\Lambda$ is such that $M^T=M$ and $XM=A^T$, hence $X^TX=I$ implies that $M$ should be such that $M^TM=M^2=AA^T$.

Using [pseudo-inverse matrices](http://fr.wikipedia.org/wiki/Pseudo-inverse) if $A$ is not invertible and the usual definition of the square root of a symmetric matrix, one sees that the maximizing and minimizing matrices $X$ and the maximum and minimum values of $F$ are 

When $A$ is invertible, one can use the usual definition of the square root of a symmetric matrix to find $M$. The maximizing and minimizing matrices $X$ and the maximum and minimum values of $F$ are$$
X\_0=A^TM^{-1}=\pm A^T(AA^T)^{-1/2},\quad
F(X\_0)=\pm\mbox{trace}((AA^T)^{1/2}).
$$
When $A$ is not invertible, the formula $X\_0=A^TM^{-1}$, whichever meaning one gives to the notation $M^{-1}$, cannot yield an orthogonal matrix $X\_0$ since, if $A$ is not invertible, neither are $A^T$, nor $A^T$ times any matrix. On the other hand, the formula $F(X\_0)=\pm\mbox{trace}((AA^T)^{1/2})$ might still be valid.
Another way to get did's answer is to use the [SVD](http://en.wikipedia.org/wiki/Singular_value_decomposition) to reduce to the case of $A$ being diagonal: We have $A = USV'$, where $U$ and $V$ are orthogonal and $S$ is diagonal, and so 

$$\def\Tr{\operatorname{Tr}}\Tr(AX) = \Tr(SV'XU)\;.$$

Since $X\to V'XU$ is a bijection on the set of orthogonal matrices, 

$$\min\{\Tr(AX)\mid X\text{ orthogonal}\} = \min\{\Tr(SW)\mid W\text{ orthogonal}\}\;.$$

But $\Tr(SW) = \sum\_iS\_{ii}W\_{ii}$, and all the $S\_{ii}$ are non-negative and $-1\le W\_{ii}\le1$, so that $\Tr(SW)\ge\Tr(S(-I))$, and so the minimum occurs at $W=-I$ (or $X=-VU'$) and is $-\sum\_iS\_{ii}$.