If $(A-B)^2=AB$, prove that $\det(AB-BA)=0$. 
Let $A,B\in M_{n}(\mathbb{Q})$. If $(A-B)^2=AB$, prove that $\det(AB-BA)=0$.

I considered the function $f:\mathbb{Q}\rightarrow \mathbb{Q}$, $f(x)=\det(A^2+B^2-BA-xAB)$ and I obtained that:
$$f(0)=\det(A^2+B^2-BA)=\det(2AB)=2^n\det(AB)$$
$$f(1)=\det(A^2+B^2-BA-AB)=\det((A-B)^2)=\det(AB)$$
$$f(2)=\det(A^2+B^2-BA-2AB)=\det((A-B)^2-AB)=\det(AB-AB)=0$$
I don't have any other idea. 

The idea is to use $(-3\pm\sqrt 5)/2$.  We begin with
$$\begin{align}
(A+xB)(A+x'B) &= A^2 + x BA + x' AB + xx' B^2 \\ &= A^2 + (x' + x) AB+x(BA-AB)+xx'B^2.\end{align}
$$
Now we find numbers $x$, $x'$ such that $x'+x  = -3$, $xx'=1$. 
These numbers are the roots of the quadratic equation 
$$
\lambda^2 +3\lambda +1 = 0.
$$
Thus, we have $x = \frac{-3+\sqrt 5}2$ and $x'= \frac{-3-\sqrt 5}2$. 
With these, we have
$$\begin{align}
(A+xB)(A+x'B) &= A^2 -3AB + B^2 + x(BA-AB) \\ &= BA-AB + x(BA-AB) = (BA-AB)(1+x).\end{align}
$$
We now have that
$$
\det(A+xB)(A+x'B) =q \in \mathbb{Q},
$$
and 
$$
q=(1+x)^n \det(BA-AB).
$$
Then it follows by $A, B\in M_n(\mathbb{Q})$ and $(1+x)^n\in\mathbb{R} \backslash \mathbb{Q}$ that $\det(BA-AB)=0$.  

An example with $AB\neq BA$
For the example that @Jonas Meyer requested, here it is:
$$
A=\begin{pmatrix} 1 & 1 \\ 0 & \frac{-3+\sqrt 5}2\end{pmatrix}, \ \ B=\begin{pmatrix} \frac{3+\sqrt 5}2 & 1 \\ 0 & -1\end{pmatrix}.
$$
Then 
$$
A-B= \begin{pmatrix} \frac{-1-\sqrt 5}2 & 0 \\ 0 & \frac{-1+\sqrt 5}2\end{pmatrix},$$
$$
(A-B)^2 = \begin{pmatrix} \frac{3+\sqrt 5}2 & 0 \\ 0 & \frac{3-\sqrt 5}2\end{pmatrix} = AB.
$$
But, 
$$
BA =\begin{pmatrix} \frac{3+\sqrt 5}2 & \sqrt 5 \\ 0 & \frac{3-\sqrt 5}2\end{pmatrix} \neq AB.
$$
