Matrix transformation shearing the triangle to be right triangle If we have a triangle at $(1,1), (5,3), (7,1)$, how to find the sheared matrix to transform the triangle to be right triangle at $(1,1)$.
Is it that we need to find $i$ in
$\begin{pmatrix}
1 && i \\
0 && 1 \\
\end{pmatrix}$
But how to ensure the right triangle.

Definition
Coordinate Transformation
Given a 2D Point $(x,y)$, to transform the point to another 2D coordinate space is defined through the following set of equations
$$x'=a\cdot x + b\cdot y$$
$$y'=c\cdot c + d\cdot y$$
where $a,b,c,d$ are real value constant with the constraint $a\cdot d - b\cdot c \ne 0$
And can be written in the matrix form as
$$\begin{pmatrix}
x' \\
y' \\
\end{pmatrix}
=
\begin{pmatrix}
a && b \\
c && d \\
\end{pmatrix}
\begin{pmatrix}
x \\
y \\
\end{pmatrix}
$$
or in short
$$T( \vec x ) = \mathbf{A} \vec x $$
where $\vec x$ is a column vector.
For multiple vectors, the same can be written as
$$T( \vec X ) = \mathbf{A} \vec X \tag1$$
where
$$T(\vec X) = \left [\vec x_1\,  \vec x_2\,  ....\,  \vec x_n  \right ]$$
In your case
$$A=\begin{pmatrix}
1 && i \\
0 && 1 \\
\end{pmatrix}$$
$$X=\begin{pmatrix}
1 && 5 && 7 \\
1 && 3 && 1\\
\end{pmatrix}$$
Representing as $(1)$ we get
$$T(\vec X)=\begin{pmatrix}
1 && i \\
0 && 1 \\
\end{pmatrix}
\begin{pmatrix}
1 && 5 && 7 \\
1 && 3 && 1\\
\end{pmatrix}$$
$$\Rightarrow T(\vec X)=\begin{pmatrix}i + 1 & 3 i + 5 & i + 3\\1 & 3 & 1\end{pmatrix}\tag2$$
$$\Rightarrow T(\vec X)=\left [ \vec x_1\,\vec x_2\,\vec x_3\right ]$$
Once we determine the transposed vector, we need to determine the dot product of the respective vectors to determine, which of them equates to 0 satisfy the condition where in one of the points is $(1,1)$
$$\vec x_1 \cdot \vec x_2 = \left(i + 1\right) \left(3 i + 5\right) + 3$$
Solving which gives
$$\begin{bmatrix}- \frac{4}{3} - \frac{2}{3} \sqrt{2} \mathbf{\imath}, & - \frac{4}{3} + \frac{2}{3} \sqrt{2} \mathbf{\imath}\end{bmatrix}$$
$$\vec x_2 \cdot \vec x_3 = \left(i + 3\right) \left(3 i + 5\right) + 3$$
Solving which gives
\begin{bmatrix}- \frac{7}{3} - \frac{1}{3} \sqrt{5} \mathbf{\imath}, & - \frac{7}{3} + \frac{1}{3} \sqrt{5} \mathbf{\imath}\end{bmatrix}
$$\vec x_3 \cdot \vec x_1 = \left(i + 1\right) \left(i + 3\right) + 1$$
Solving which gives
\begin{bmatrix}-2\end{bmatrix}
Replacing each of these values in $(2)$, its evident that only the last solution satisfies the conditions
\begin{pmatrix}-1 & -1 & 1\\1 & 3 & 1\end{pmatrix}
So $i=-2$ is the solution to the problem
