Factorize $abx^2-(a^2+b^2)x+ab$ Factorize the quadratic trinomial $$abx^2-(a^2+b^2)x+ab.$$
The discriminant of the trinomial is $$D=(a^2+b^2)^2-(2ab)^2=\\=(a^2+b^2-2ab)(a^2+b^2+2ab)=(a-b)^2(a+b)^2=\\=\left[(a-b)(a+b)\right]^2=(a^2-b^2)^2\ge0 \text{ } \forall a,b.$$
So the roots are $$x_{1,2}=\dfrac{a^2+b^2\pm\sqrt{(a^2-b^2)^2}}{2ab}=\dfrac{a^2+b^2\pm\left|a^2-b^2\right|}{2ab}.$$ How can I expand the modulus here? Have I calculated the discriminant in a reasonable way? Can we talk about "the discriminant of a quadratic trinomial" or only the corresponding quadratic equation (the trinomial=0) has a discriminant? What about "the roots of a trinomial" (or of the corresponding quadratic equation)?

Using the quadratic formula is a good approach, and everything you have done so far is correct. The modulus/absolute value can be replaced by normal brackets as there is a $\pm$ preceding it, and the numbers involved here are real. Notice that:
$$\frac {a^2 +b^2 + (a^2 - b^2)}{2ab} = \frac ab, \quad \frac {a^2 +b^2 - (a^2 - b^2)}{2ab} = \frac ba$$
hence the roots are $\dfrac ab$ and $\dfrac ba$, and we are looking at the factors $(bx- a)$ and $(ax - b)$. Now:
$$(bx-a)(ax- b) = abx^2 - a^2 x - b^2 x + ab$$
which is exactly what we are trying to factorize.
