Congruence and diagonalizations How does one find matrix $M\in M_3(\mathbb R)$ such that $M^TAM=I$ where $$A=\left(\begin{array}{rrr}2&0&0\\ 0&3&-1\\ 0&-1&3\end{array}\right)$$
and simultaneously, $M^TBM$ is some diagonal matrix where $$B=\left(\begin{array}{rrr}1&3&-3\\ 3&3&1\\ -3&1&3\end{array}\right)$$?
Thanks.

As noted in my answer to your related question, there is a theorem regarding simultaneous diagonalization by congruence: if $A,B$ are real symmetric and $A$ is nonsingular, then they are simultaneously congruent to diagonal matrices if and only if $C=A^{−1}B$ is diagonalizable by similarity transform (Horn and Johnson, Matrix Analysis, Theorem 4.5.15.).
When all eigenvalues of $C$ are distinct, the matrix for diagonalizing $C$ via similarity transform is by itself the matrix for simultaneous diagonalizing $A$ and $B$ by congruence. Unfortunately, in your problem, $C$ has repeated eigenvalues and hence some additional manual work is needed. Specifically, we first try to diagonalize $C$ and obtain
$$
C = A^{-1}B =
\begin{pmatrix}
1/2&3/2&-3/2\\
3/4&5/4&3/4\\
-3/4&3/4&5/4
\end{pmatrix}
=M_1\ D\ M_1^{-1},
$$
where
$$
M_1 =
\begin{pmatrix}
2&1&0\\
-1&1&1\\
1&0&1
\end{pmatrix},\quad D=\textrm{diag}(-1,2,2).
$$
So, our first candidate for $M$ is $M_1$. However, observe that $D$ has repeated eigenvalues. So there is a chance that $M_1^\top AM_1$ or $M_1^\top BM_1$ are not diagonal. Indeed,
$$
M_1^\top AM_1 =
\begin{pmatrix}
16&0&0\\
0&5&2\\
0&2&4
\end{pmatrix},
\quad M_1^\top BM_1 =
\begin{pmatrix}
-16&0&0\\
0&10&4\\
0&4&8
\end{pmatrix}.
$$
So we need to further diagonalize $\begin{pmatrix}5&2\\2&4\end{pmatrix}$ and  $\begin{pmatrix}10&4\\4&8\end{pmatrix}$ simultaneously via congruence. Since the two matrices are multiples of each other and they are symmetric, it suffices to find a matrix $M_2$ that orthogonally diagonalizes either one of them. It is not difficult to work out such a matrix: for
$$
\lambda=\frac{9+\sqrt{17}}{2},\quad
M_2=\frac{1}{\sqrt{\lambda+4}}\begin{pmatrix}\lambda-4&-2\\2&\lambda-4\end{pmatrix},
$$
we have
$$
\begin{pmatrix}5&2\\2&4\end{pmatrix} = M_2\begin{pmatrix}\lambda&0\\0&9-\lambda\end{pmatrix}M_2^\top.
$$
Now we may set
$$
M = M_1\begin{pmatrix}1\\&M_2\end{pmatrix}.
$$
For this particular problem, since $A$ is positive definite, there is a simpler solution -- find the square root of $A$, then orthogonally diagonalize $(\sqrt{A}^\top)^{-1}B\sqrt{A}^{-1}$. In other words, if $\sqrt{A}$ is a matrix such that $\sqrt{A}^\top\sqrt{A}=A$, and $Q$ is an orthogonal matrix such that $Q^\top(\sqrt{A}^\top)^{-1}B\sqrt{A}^{-1}Q$ is diagonal, then you may set $M=\sqrt{A}^{-1}Q$. In this case you will get $M^\top AM=I$.
