My question is, why, of all possible combinations of final velocities that conserve momentum, does this one lead to the greatest loss in kinetic energy?
There's a cute geometric interpretation that I want to show.
We'll consider collisions in one dimension. Let $m_1$, $m_2$ be the masses of the two objects; $v_1$, $v_2$ be the velocities.
The velocities of the two objects can be represented, through some modification, as points in a plane. Since
$$\text{KE}=\sum_{i=1}^N\frac{1}{2}m_iv_i^2$$
we might become inspired to plot our points as
$$(x, y) = (\sqrt{m_1}v_1, \sqrt{m_2}v_2)$$
Now if we consider a specific snapshot in time, the set of all points which have a constant KE is defined by a circle:
$$\text{KE}=\frac{1}{2}(x^2+y^2)$$
centered around the origin, with radius $\sqrt{2\,\text{KE}}$.
Likewise, since conservation of momentum is defined as
$$m_1v_1 + m_2v_2 = \text{const.}$$
the set of all points that have a constant momentum is defined by a line:
$$y = -\sqrt{\frac{m_1}{m_2}}(x-x_0) + y_0$$
$$\rule[0]{200pt}{0.4pt}$$
If we imagine the initial velocities to be a pre-existing point on the plane, a collision is represented as a discontinuous hiccup. Conservation of kinetic energy as a circle and conservation of momentum as a line will, in most cases, delineate one other intersection. 512
This intersection denotes an elastic collision.
However, if kinetic energy is lost, as in an inelastic collision, then we are confined to a disk within the plane. Amalgamation with momentum conservation imprisons us to a line segment, depicted below in blue.
Where on this line segment do we lose the most kinetic energy? The radius of our $\text{KE}$ circle was defined by $\sqrt{2\,\text{KE}}$, which means that we want our new point, while still on the conserved momentum segment, to be as close as possible to the origin.
In other words, if we drew a line from the origin to the most inelastic point, it would be perpendicular to the momentum line.
But the slope of the momentum line is already given as $-\sqrt{m_1/m_2}\,$. So the most inelastic line must have the equation
$$\begin{align} y&=\sqrt{\frac{m_2}{m_1}}x \ \sqrt{m_2}v_2 &= \sqrt{\frac{m_2}{m_1}}\sqrt{m_1}v_1 \ v_2 &= v_1 \ & \qquad \qquad \qquad \qquad \blacksquare \end{align}$$
Note that this theorem is invariant of initial conditions, as expected.
$\rule[0.1pt]{300pt}{0.4pt}$
To connect this answer to the discussion on changing reference frames, simply note that a change in frame corresponds to translations of the origin about the $\sqrt{m_2/m_1}$ line; the line of perfect inelasticity. When the origin happens to land on the momentum line, this frame is the center-of-mass frame.
This interpretation was inspired by a youtube video by 3Blue1Brown. 512