Find all matrices $X$ such that $ABXB^tA^t=I$ Find all matrices $X$ such that: 
$$ABXB^tA^t=I$$ if 
$A=\begin{pmatrix}
1 &-2 &2\\
3 &-5 &6\\
-1 &2 &-1
\end{pmatrix}$ and $B=\begin{pmatrix}
-3 &-2 &-2\\
2 & 1 &1\\
6 &3 &4
\end{pmatrix}$. 
So I managed to get that $AB=\begin{pmatrix}
5 &2 &4\\
17 &7 &13\\
1&1&0
\end{pmatrix}$ and  $B^tA^t=(AB)^t=\begin{pmatrix}
5 &17 &1\\
2 &7 &1\\
4 &13 &0
\end{pmatrix}$
So we have $\begin{pmatrix} 
5 &2 &4\\
17 &7 &13\\
1&1 &0
\end{pmatrix}\cdot X\cdot\begin{pmatrix}
5 &17 &1\\
2&7&1\\
4&13&0
\end{pmatrix}=I$  
Now how do I get $X$ from here?

Matrices multiplication is not commutative. You are allowed to multiply an equation by a non-zero matrix, but you have to mind the side from which you are applying the multiplication.
Let us say that $P=AB$ and $Q=B^TA^T$.
Your equation takes the form of $P \cdot X \cdot Q = I$.
You can multiply each side by $P^{-1}$ from the left and $Q^{-1}$ from the right to get (note that $\det(P)\ne0\ne\det(Q)$, which makes $P$ and $Q$ invertible):
$A^{-1} \cdot P \cdot X \cdot Q \cdot Q^{-1} = P^{-1} \cdot I \cdot Q^{-1}$
This can be simplified based on $M^{-1} \cdot M = M \cdot M^{-1} = I$:
$I \cdot X \cdot I = P^{-1} \cdot I \cdot Q^{-1}$
This can further be simplified based on $M \cdot I = M$:
$X=P^{-1} \cdot Q^{-1}$
Mind that:
$M^{-1}=\frac{1}{\det(M)}\left(M^*\right)^T$
Where $M^*$ is the cofactor matrix.
