main axis transformation with the conic $Γ:−6x^2−6yx+4x+2y^2−4y+1=0$ The purpose of this exercise is to reduce the conic  $Γ:−6x^2−6yx+4x+2y^2−4y+1=0$   to the canonical expression.
What I already have is the Eigenvectors $-7,3$ and the eigenvectors $\begin{pmatrix} 3 \\ 1 \end{pmatrix}, \begin{pmatrix} -1 \\ 3 \end{pmatrix}$. This gives me the rotation matrix $\begin{pmatrix} 3/\sqrt(10) & -1/\sqrt(10) \\ 1/\sqrt(10) & 3/\sqrt(10) \end{pmatrix}$.
After that my next step, determining the mixed term after rotation, I fail. The values before the $x^2,y^2$ are the Eigenvalues and the values before $x,y$ are $$\begin{pmatrix} 4 & -4 \end{pmatrix} \begin{pmatrix} 3/\sqrt(10) & -1/\sqrt(10) \\ 1/\sqrt(10) & 3/\sqrt(10) \end{pmatrix}= \begin{pmatrix} 4 \sqrt{\dfrac{2}{5}} & -8\sqrt{\dfrac{2}{5}} \end{pmatrix}.$$ This gives me $$3y^2-7x^2+4 \sqrt{\dfrac{2}{5}}x-8 \sqrt{\dfrac{2}{5}}y+1.$$
But this is wrong. Why?

The given conic equation is of the form
$ r^T Q r + r^T b + c = 0 $
with
$Q = \begin{bmatrix} -6 && - 3 \\ - 3 && 2 \end{bmatrix} $
$ b = [ 4, -4]^T $
$ c = 1 $
From $Q$ we determine the rotation angle $\theta $ using the formula
$ \theta = \dfrac{1}{2} \tan^{-1} \left( \dfrac{ 2 Q_{12} }{Q_{11} - Q_{22}} \right) = \dfrac{1}{2} \tan^{-1} \left( \dfrac{3}{4} \right) $
The rotation matrix whose columns are the unit eigenvectors is
$ R = \begin{bmatrix} \cos \theta && -\sin \theta \\ \sin \theta && \cos \theta \end{bmatrix} $
Since $ \tan(2 \theta) = \dfrac{3}{4} $ then $\cos(2 \theta) = \dfrac{4}{5} $
Hence $\cos(\theta) = \sqrt{ \dfrac{1 + \cos(2\theta)}{2} } = \dfrac{3}{\sqrt{10} } $ and $ \sin(\theta) = \dfrac{1}{\sqrt{10}} $, therefore,
$R = \dfrac{1}{\sqrt{10}} \begin{bmatrix} 3 && -1 \\ 1 && 3 \end{bmatrix}$
With this $R$, we can diagonalize $Q$ as follows
$ Q = R D R^T $
The diagonal entries on the diagonal matrix $D$ are given by
$D_{11} = \dfrac{1}{2} \big( Q_{11} + Q_{22} \big) + \dfrac{1}{2} \big( Q_{11} - Q_{22} \big) \cos(2 \theta) + Q_{12} \sin(2 \theta) $
$D_{22} = \dfrac{1}{2} \big( Q_{11} + Q_{22} \big) - \dfrac{1}{2} \big( Q_{11} - Q_{22} \big) \cos(2 \theta) - Q_{12} \sin(2 \theta) $
And these evaluate to
$ D_{11} = \dfrac{-4}{2} + \dfrac{-32}{10} + \dfrac{-9}{5} = - 2 - 3.2 - 1.8 = -7 $
$D_{22} = \dfrac{-4}{2} - \dfrac{-32}{1} - \dfrac{-9}{5} = -2 + 3.2 + 1.8 = 3 $
i.e.
$D = \begin{bmatrix} -7 && 0 \\ 0 && 3 \end{bmatrix}$
Now the equation of the conic is
$ r^T R D R^T r + r^T b + c = 0$
Let $ w = R^T r $ (from which it follows that $r = R w$ ), then
$ w^T D w + w^T R^T b + c = 0 \hspace{40pt}(*)$
The vector $R^T b$ is given by
$R^T b = \dfrac{1}{\sqrt{10}} \begin{bmatrix} 3 && 1 \\ -1 && 3 \end{bmatrix} \begin{bmatrix} 4 \\ - 4 \end{bmatrix} = \dfrac{1}{\sqrt{10}} \begin{bmatrix} 8 \\ -16 \end{bmatrix}$
So, in the transformed coordinate $w = [x, y]$ the equation is
$ - 7 x^2 + 3 y^2 + \dfrac{8}{\sqrt{10}} x - \dfrac{16}{\sqrt{10}} y + 1 = 0$
