Find all matrices $X$ such that $ABXB^tA^t=I$ Find all matrices $X$ such that: 
$$ABXB^tA^t=I$$ if 
$A=\begin{pmatrix}
1 &-2 &2\\
3 &-5 &6\\
-1 &2 &-1
\end{pmatrix}$ and $B=\begin{pmatrix}
-3 &-2 &-2\\
2 & 1 &1\\
6 &3 &4
\end{pmatrix}$. 
So I managed to get that $AB=\begin{pmatrix}
5 &2 &4\\
17 &7 &13\\
1&1&0
\end{pmatrix}$ and  $B^tA^t=(AB)^t=\begin{pmatrix}
5 &17 &1\\
2 &7 &1\\
4 &13 &0
\end{pmatrix}$
So we have $\begin{pmatrix} 
5 &2 &4\\
17 &7 &13\\
1&1 &0
\end{pmatrix}\cdot X\cdot\begin{pmatrix}
5 &17 &1\\
2&7&1\\
4&13&0
\end{pmatrix}=I$  
Now how do I get $X$ from here?

If either $AB$ or $B^TA^T$ is singular, there is no chance to get a solution. 
Therefore the $3 \times 3$ matrices, have to be nonsingular (and they are, I just checked). 
So, to get $X$ you need to invert both of them and get to $X=B^{-1}A^{-1}A^{-T}B^{-T} = B^{-1}(A^TA)^{-1}B^{-T}$
Usually you would decompose $AB$ into $LR$ or $QR$ decompositions and pull them to the right hand side. 
