When is $n\cdot m\cdot [(n-8)\cdot m+4]-4$ an integer square? For which odd integers $n$ and $m$, with $n \ge m$ is $$n\cdot m\cdot [(m-8)\cdot n+4]-4$$ a square?
Part of my efforts so far:
The product of the three factors above is a sum of two squares (the square we wish to find plus $2^2$), thus all three factors are a sum of two squares too (since their prime factorization is of the right form).
But that is not enough, as their product minus 4 needs to be a square too.
So $n=17$ and $m=13$ look to be OK at first sight, as the third factor then is $89=8^2+5^2$, but the product $13\cdot 17\cdot 89$ is not a square plus 4.
On the other hand, $n=17$ and $m=17$ is OK, as the third factor now is $157=11^2+6^2$ and also $17\cdot 17 \cdot 157=213^2+2^2$.
Taking the sum of two squares to be the determinant of a $2x2$ matrix
$$\left(\begin{array}{cc}a&b\\-b&a \end{array}\right)$$
it is easy to see that $n$, $m$ and $(m-8)\cdot n+4$ then correspond to matrices whose product is a matrix as above with $b=\pm2$.
For example
$$\left(\begin{array}{cc}4&1\\-1&4 \end{array}\right)\cdot \left(\begin{array}{cc}4&1\\-1&4 \end{array}\right)\cdot \left(\begin{array}{cc}11&-6\\6&11 \end{array}\right)=\left(\begin{array}{cc}213&-2\\2&213 \end{array}\right)$$
Not sure if this helps finding the solution, though.
A couple of the solutions I found are
$$
\begin{array}{l|l|c|l}
\text{n} & \text{m} & (m-8)\cdot n+4 & \text{product} \\
\hline
17=4^2+1^2 & 17=4^2+1^2 & 157=11^2+6^2 & 45373=213^2+2^2 \\
29=5^2+2^2 & 13=3^2+2^2 & 149=10^2+7^2 & 56173=237^2+2^2 \\
3277=54^2+19^2 & 17=4^2+1^2 & 29497=171^2+16^2 & 1643248373=40537^2+2^2 \\
\end{array}
$$
What I would like to have is a general expression for all solutions, or an algorithm to find the next solution.

For every $m$ such that $-1$ is quadratic residue modulo $m$ and $m>5$ you will have infinitely many $n$  which satisfy that $n m ((m-8) n+4)-4$ is perfect square.
For Example: for $m=13$ we get that $n=\{29,125,1956797,8346845,130248348509,555582723389,8669590571525885,3698069722589
   8397,577065287491657635485,2461509168194666404349,38410619657350124961653309,1
   63842973216392687980406365,\cdots\}$
Note that the above sequence follow an explicit formula, so for any fixed $m$ which fulfill the conditions above, you can have a general formula, but it will be correct just for the fixed $m$, a general formula for $n,m$ without any one be fixed might be hard to come by if not impossible.
