$(a+b)^2+4ab$ and $a^2+b^2$ are both squares I cannot find a complete answer to the following problem (this is the source):

Q.  Find all positive integers $(a,b)$ for which $(a+b)^2+4ab$ and $a^2+b^2$ are both squares.


Just something: clearly if $(a,b)$ works then $(a/c,b/c)$ works as well, where $c$ is the greatest common divisor of $(a,b)$. Hence we can assume they are coprime. 
List of known primitive solutions: As remarked by Michael below, this is equivalent to solve the equation $(x^2-y^2)^2+12xy(x^2-y^2)+4x^2y^2=z^2$ with positive integers $x,y$. Adding the constraint $\mathrm{gcd}(x,y)=1$, by computer calculations we can see that all solutions $(x,y)$ with $x,y \le 30000$ are only the following ones: $(3,2)$, $(5,1)$, $(7,85)$, $(39,46)$, $(2717,1380)$, $(4097, 1337)$. This leads to the primitive solutions
$$
(a,b)=(5,12),
$$ 
$$
(a,b)=(5477689,7498920).
$$ 

Edit (05 Dec 2015): the ones above are the unique primitive solutions also with the (last new) constraint $x,y \le 80000$..

Let $q^2 = (a+b)^2$, $p^2 = a^2 + b^2$ and $r^2 = (a+b)^2 + 4ab$. Note that $r^2 + p^2 = 2q^2$. Now let $X = r+p$, $Y = r - p$ and $Z = 2q$.
You can easily show that $r^2 + p^2 = 2q^2$ if and only if $X^2 + Y^2 = Z^2$.
So, each solution can be found as follows:


*

*Pick a Pythagorean triple $(X, Y, Z)$ in which $Z$ is even.

*Let $q = \frac{Z}{2}$, $r = \frac{X+Y}{2}$ and $p = \frac{X-Y}{2}$.

*Check if there is any Pythagorean triple $(a', b', p)$ for all $a', b'$ that $a'+b' = q$.


Hope this helps :)
