In a right-angled triangle, we can find a square in two different ways, whose vertices lie on the perimeter of the triangle. Determine the area of the common part of these squares, given the legs of the triangle.
Let the given triangle be $ABC$, where $AB$ is the hypotenuse, and the sides are conventionally denoted as $a, b, c$. For every inscribed square, two vertices must lie on one of the triangle's sides. If we look for these on the leg $CA$, then one of them will be $C$, otherwise, there would not be a square vertex on $CB$, since $AB$ can only have one vertex. Let's denote this inscribed square as $CDEF$, with $D$ on $CA$, and thus $F$ on $CB$.

![](https://cdn.mathpix.com/cropped/2024_05_02_005836640fc03ad5f974g-1.jpg?height=789&width=1019&top_left_y=332&top_left_x=542)

In this inscribed square, the diagonal $CE$ bisects the right angle $ACB$, so

$$
BE: EA = CB: CA = a: b
$$

and

$$
BE: (BE + EA) = BE: c = a: (a + b)
$$

from which $BE = \frac{ac}{a + b}$.

Thus, from the similarity of triangles $BEF$ and $BAC$, the side of the square is

$$
FE = \frac{b}{c} BE = \frac{ab}{a + b}
$$

If we want two vertices of the square on the hypotenuse $AB$ - let these be $H$ and $I$ - these are separated by the altitude $CC'$, since the endpoints of the square side $GJ$ parallel to $HI$ cannot be on the same leg. Let $H$ and $G$ be on the side of $CC'$ containing $A$, and let $IJ = IH = y$.

The extreme segments of the hypotenuse, divided into three parts, using similar triangles,

$$
BI = \frac{a}{b} y, \quad AH = \frac{b}{a} y
$$

and adding $IH$ to these, we get the hypotenuse $AB$ of the triangle, from which

$$
y = \frac{c}{\frac{a}{b} + 1 + \frac{b}{a}} = \frac{abc}{a^2 + ab + b^2}
$$

The vertex $G$ is on the segment $CD$ for any value of the ratio $a/b$, because

$$
CG = \frac{b}{c} JG = \frac{ab^2}{a^2 + ab + b^2} = CD \cdot \frac{ab + b^2}{a^2 + ab + b^2} < CD
$$

Similarly, $J$ is a point on the segment $CF$. Furthermore, $I$ is always on the segment $BE$ and $H$ is on the segment $AE$, because

$$
\frac{BI}{BE} = \frac{a(a + b)}{a^2 + ab + b^2} < 1 \quad \text{and} \quad \frac{AH}{AE} = \frac{b(a + b)}{a^2 + ab + b^2} < 1
$$

Accordingly, the common part of the two squares is the pentagon $ELJGM$, where $L$ is the intersection point of the segments $EF$ and $IJ$, and $M$ is the intersection point of the segments $ED$ and $HG$: from the second square, we need to exclude the triangles $EIL$ and $EHM$.

The legs of these triangles, using the previous partial results,

$$
\begin{gathered}
EI = BE - BI = \frac{ab^2c}{(a + b)(a^2 + ab + b^2)} = \frac{b}{a + b} \cdot y \\
IL = \frac{a}{b} EI = \frac{a}{a + b} \cdot y
\end{gathered}
$$

and similarly, $HE = IL$ and $HM = IE$.

Based on these, the sought area is

$$
y^2 - EI \cdot IL = y^2 \left(1 - \frac{ab}{(a + b)^2}\right) = \frac{a^2 + ab + b^2}{(a + b)^2} \cdot y^2 = \frac{a^2 b^2 (a^2 + b^2)}{(a + b)^2 (a^2 + ab + b^2)}
$$

Remark. If we consider the hypotenuse and one of the acute angles as given instead of the legs, then the area of the common part of the two squares is

$$
\frac{c^2}{2} \cdot \frac{\sin^2 2\alpha}{(1 + \sin 2\alpha)(2 + \sin 2\alpha)} \leqq \frac{c^2}{12}
$$

The upper limit is reached by the area when $\alpha = 45^\circ$, and in this case, it is $\frac{1}{3}$ of the area of the original triangle.