Diagonalization of a quadratic form, such as $\lambda_i=\{-1,1,0\}$ I am given a quadratic form 
$$Q(x) = x_1^2 + 5 x_2^2 + 3 x_3^2 - 8 x_1 x_3 + 8 x_2 x_3$$ 
and what I need is to diagonalization of a quadratic form, such as $\lambda_i=\{-1,1,0\}$ using the Lagrange method (could not find the term for that in English).
So I am trying to extract square terms and end up with a canonical quadratic form such as $Q(y)=3y_1^2-\frac{13}{3}y_2^2+y_3^2$ where I use the following substitutions
$$
\begin{cases} y_1 = -\frac{4}{3}x_1+\frac{4}{3}x_2+x_3 \\ y_2 =x_1-\frac{16}{13}x_2 \\ y_3 = 9x_2 \end{cases}
$$
Which gives us a transformation of the type $Yx=y$ where the matrix Y is:
$$
\begin{pmatrix} -\frac{4}{3} & \frac{4}{3} & 1 \\ 1 & -\frac{16}{13} & 0 \\ 0 & 9 & 0 \end{pmatrix}
$$
Then we make the following substitutions:
$$
\begin{cases} z_1 = -\frac{1}{\sqrt3}y_1 \\ z_2 =\sqrt\frac{3}{13}y_2 \\ z_3 = y_3 \end{cases}
$$
Which gives us a matrix $Z$, which we use for transformation $y=Zz$.
As a result we have equation $Yx=y$ and $y=Zz$, so the matrix of transformation from the initial quadratic form to the diagonal form with $\lambda=\{-1,1,0\}$ will be the matrix $Y^{-1}Z$
Inverse matrixes give
$$
Y^{-1}=\begin{pmatrix} -1 & 1 & \frac{16}{117} \\ 0 &0& -\frac{1}{9} \\ 0 & \frac{4}{3} & \frac{4}{117} \end{pmatrix}
$$
$$
Z=\begin{pmatrix} \sqrt3 & 0 & 0 \\ 0 &\sqrt\frac{13}{3}& 0 \\ 0 & 0 & 1 \end{pmatrix}
$$
Giving me as the end result 
$$
\begin{pmatrix} -\sqrt3 & \sqrt\frac{13}{3} & \frac{16}{117} \\ 0 &0& \frac{1}{9} \\ 0 & \sqrt\frac{208}{27} & \frac{4}{117} \end{pmatrix}
$$
Is my reasoning correct and if so, is there a better way to do this task?

Your method seems to be less efficient than a faster method I know; I can't vouch for the accuracy of your method if you don't give a final answer.
The correct way to do this is to write
$$\mathbf v = \begin{pmatrix} x_1\\x_2\\x_3 \end{pmatrix},$$
and then write the quadratic form as
$$Q(x) = \mathbf v^\intercal A \mathbf v,$$
for an appropriate symmetric matrix $A$, which in this case is
$$\begin{pmatrix}1&0&-4\\0&5&4\\-4&4&3\end{pmatrix}.$$
Since this is symmetric, we can diagonalise it, writing $A = PDP^{-1}$, where $P$ is the matrix of eigenvectors, and $D$ is a diagonal matrix of eigenvalues. Multiplying $P$ and $P^{-1}$ beside $\mathbf v$ as well will give you your required form.
