5. Let the sequences $\left\{a_{n}\right\}$ and $\left\{b_{n}\right\}$ satisfy $a_{0}=2, b_{0}=2$,
and $\left\{\begin{array}{l}a_{n+1}=a_{n} \sqrt{1+a_{n}^{2}+b_{n}^{2}}-b_{n}, \\ b_{n+1}=b_{n} \sqrt{1+a_{n}^{2}+b_{n}^{2}}+a_{n} .\end{array}\right.$
Then $a_{2017}^{2}+b_{2017}^{2}=$ $\qquad$
5. $3^{2^{2018}}-1$.

Notice,
$$
\begin{array}{l} 
1+a_{n+1}^{2}+b_{n+1}^{2} \\
= 1+a_{n}^{2}\left(1+a_{n}^{2}+b_{n}^{2}\right)+b_{n}^{2}+ \\
\quad b_{n}^{2}\left(1+a_{n}^{2}+b_{n}^{2}\right)+a_{n}^{2} \\
=\left(1+a_{n}^{2}+b_{n}^{2}\right)^{2} .
\end{array}
$$

Then $1+a_{n}^{2}+b_{n}^{2}=3^{2^{n+1}}$
$$
\Rightarrow a_{2017}^{2}+b_{2017}^{2}=3^{2^{2018}}-1
$$