Find $(a^2+b^2),$ where $(a+b)=\dfrac{a}{b}+\dfrac{b}{a}$ The question is the same .Find $a^2+b^2$. I think we have to find $a$ and $b$ firstly. Given that $a$ and $b$ are integers.

I assume your formula is $a+b=\dfrac ab+\dfrac ba,$ then $a^2+b^2=ab(a+b)$ and since $a^2+b^2=(a+b)^2-2ab,$ we have $$x^2-2y=xy,$$ where $x=a+b$ and $y=ab.$ From here you can find $x$ in-terms of $y$ as $$x=\dfrac{y\pm\sqrt{y^2+8y}}{2}.$$ Then $$\color{Green}{a^2+b^2=\left(\dfrac{y\pm\sqrt{y^2+8y}}{2}\right)y},$$ where $y$ is a parameter.  

Also, if you only interest on $a,b$ integer solutions, we need to find $y$ so that $y^2+8y$ is a perfect square.

Edit:
To do this, note that $z^2=y^2+8y=(y+4)^2-16$ implies $16=(y-z+4)(y+z+4).$ Now find the all possible values of $y-z+4$ and $y+z+4$ using factors of $16.$ Then you will have to solve few simultaneous equations to find corresponding $y.$
Later:  

As @MANMAID pointed out $y=1$ is the only possible value and this gives us $$\color{Red}{a^2+b^2=2.}$$

