Overdetermined system with parameters $$\begin{cases}
ax & +y &=2a\\
x &+by  &= b \\
ax &+ (5-b^2)y &= 1 \tag{A,B,C (in order)}
\end{cases}$$
Where $(x,y)$ are variables and $a,b$ are constants. 


*

*What combination of $a$ and $b$ yields infinitely many solutions to our system?



By subtracting $A$ from $C$ we get:
$\begin{cases}
ax & +y &=2a\\
x &+by  &= b \\
&(5-b^2)y - y &= 1 - 2a \tag{A',B',C' (in order)}
\end{cases}$
$\iff\begin{cases}
ax & +y &=2a\\
x &+by  &= b \\
&y(4-b^2) &= 1 - 2a \tag{A'',B'',C'' (in order)}
\end{cases}$
$\iff\begin{cases}
ax & +y &=2a\\
x &+by  &= b \\
&y &= \frac{1 - 2a}{4-b^2} \tag{A''',B''',C''' (in order)}
\end{cases}$
By subtracting $C'''$ from $A'''$ we get:
$$\begin{cases}
ax &&=2a - \frac{1 - 2a}{4-b^2}\\
x &+by  &= b \\
&y &= \frac{1 - 2a}{4-b^2}
\end{cases}$$
$$\iff\begin{cases}
x &&=2 - \frac{1 - 2a}{4a-ab^2}\\
x &+by  &= b \\
&y &= \frac{1 - 2a}{(4-b^2)}
\end{cases}$$
Substituting $x$ and $y$ into $B$ gets us the following expression:
$$\left(2-\frac{1-2a}{4a-ab^2}\right) + b\cdot\left(\frac{1-2a}{4-b^2}\right) = b$$
Did I do anything right? Is this leading to the solution? Is there any more examples of this particular problem anywhere? Can't seem to find any.
NB! I ONLY SEEK THE VALUES OF $a$ AND $b$ WHEN THEY YIELD INFINITELY MANY SOLUTIONS TO OUR SYSTEM

My advice is to avoid dividing by variable expressions (since we might lose solutions or otherwise be implicitly making certain assumptions). We can row reduce the corresponding augmented matrix to get:
$$
\left[\begin{array}{cc|c}
a & 1 & 2a \\
1 & b & b \\
a & 5 - b^2 & 1
\end{array}\right]
\sim
\left[\begin{array}{cc|c}
1 & b & b \\
0 & 1-ab & 2a-ab \\
0 & 4 - b^2 & 1 - 2a
\end{array}\right]
$$
Now since the system must have infinitely many solutions, the second column cannot have a pivot. Thus, $1 - ab = 4 - b^2 = 0$. But then since the system must be consistent, the last two entries in the augmented column must be zero, so $2a - ab = 1 - 2a = 0$. Since:
$$
1 - 2a = 0 \iff \boxed{a = 1/2}
$$
we can substitute this into $1 - ab = 0$ to get that $\boxed{b = 2}$.
Indeed, this combination of $a$ and $b$ values makes the last two rows of the augmented matrix all zeroes, as desired. $~~\blacksquare$
