Rotation and surface in space Which surfaces in the space represent this equation
$$
x^2+2y^2+z^2+4xy+2xz-4yz = 1 ?
$$
I calculate the characteristic polynomial.
$$
\begin{bmatrix}
1-\lambda  & 2 &1 \\
2 & 2-\lambda &-2 \\
1 &-2 & 1-\lambda \\
\end{bmatrix}
$$
$$
A=
\begin{vmatrix}
 1& 2& 1 \\
 2& 2& -2 \\
 1& -2& 1 \\
\end{vmatrix}
$$
$$
\begin{bmatrix}
1-\lambda  & 2 &1 \\
2 & 2-\lambda &-2 \\
1 &-2 & 1-\lambda \\
\end{bmatrix}
= (1-\lambda)^2(2-\lambda)-4-4-((2-\lambda)+8(1-\lambda))
$$
$$
=(1-\lambda)^2(2-\lambda)-18-\lambda
$$
$$
=(2-\lambda)(4-\lambda)(2+\lambda)
$$
$\lambda_1 = 2$
$$
(A-\lambda I)=
\begin{vmatrix}
 -1& 2& 1 \\
 2& 0& -2 \\
 1& -2& -1 \\
\end{vmatrix}
\Rightarrow
v_1 =
\begin{bmatrix}
1\\
0\\
1\\
\end{bmatrix}
$$
$\lambda_2 = 4$
$$
(A-\lambda I)=
\begin{vmatrix}
 -3& 2& 1 \\
 2& -2& -2 \\
 1& -2& -3 \\
\end{vmatrix}
\approx
\begin{vmatrix}
 -3& 2& 1 \\
 -1& 0& -1 \\
 0& -2& -4 \\
\end{vmatrix}
\Rightarrow
v_2 =
\begin{bmatrix}
1\\
2\\
-1\\
\end{bmatrix}
$$
$\lambda_3 = -2$
$$
(A-\lambda I)=
\begin{vmatrix}
 -1& 2& 1 \\
 2& 0& -2 \\
 1& -2& -1 \\
\end{vmatrix}
\Rightarrow
v_3 =
\begin{bmatrix}
-1\\
1\\
1\\
\end{bmatrix}
$$
$$
D=
\begin{bmatrix}
2 & 0 & 0\\
0 & 4 & 0\\
0 & 0 & -2\\
\end{bmatrix}
P= \frac{1}{\sqrt{6}}
\begin{bmatrix}
\sqrt{3} & 1& -\sqrt{2}\\
0 & 2 & \sqrt{2}\\
\sqrt{3} & -1 & \sqrt{2}\\
\end{bmatrix}
$$
Where shoud i see what is this function and its rotation?

The short version is that you correctly found $P^T A P = D,$ with $P^T P = P P^T = I.$ If your new triple of coordinates is called $u,v,w,$ you now have $2 u^2 + 4 v^2 - 2 w^2 = 1,$ which is a hyperboloid of one sheet. 
