5. Function
$$
\begin{aligned}
f(x, y)= & x^{4}+y^{4}+2 x^{2} y-2 x y^{2}+3 x^{2}+ \\
& 3 y^{2}-2 x+2 y+2
\end{aligned}
$$

The minimum value of the function is
5. $\frac{9}{8}$.

Notice,
$$
\begin{array}{l} 
f(x, y) \\
= x^{4}+y^{4}+2 x^{2} y-2 x y^{2}+3 x^{2}+3 y^{2}-2 x+2 y+2 \\
= x^{4}+(2 y+3) x^{2}-2\left(y^{2}+1\right) x+y^{4}+3 y^{2}+2 y+2 \\
= x^{4}+2(y+1) x^{2}+(y+1)^{2}+y^{4}- \\
2(x-1) y^{2}+(x-1)^{2} \\
=\left(x^{2}+y+1\right)^{2}+\left(y^{2}+1-x\right)^{2} .
\end{array}
$$

In the Cartesian coordinate system, take points
$$
A\left(x, x^{2}+1\right), B\left(y^{2}+1,-y\right) \text {. }
$$

Then $f(x, y)=|A B|^{2}$.
Notice, points $A$ and $B$ lie on two parabolas, and these two parabolas are symmetric about the line $y=x$. Hence,
$$
\begin{array}{l}
|A B|_{\text {min }}=2 \min _{x} \frac{\left|x-\left(x^{2}+1\right)\right|}{\sqrt{2}} \\
=\sqrt{2} \min _{x}\left|\left(x-\frac{1}{2}\right)^{2}+\frac{3}{4}\right|=\frac{3 \sqrt{2}}{4} .
\end{array}
$$

Thus, $f(x, y)_{\text {min }}=\frac{9}{8}$.