Problem 4. Point $O$ lies inside an isosceles right triangle $A B C$. The distance from it to vertex $A$ of the right angle is 5, to vertex $B$ is 6, and to vertex $C$ is 4. Find the area of triangle $A B C$.
Answer: $13+\frac{5}{2} \sqrt{23}$.

First solution. Consider a $90^{\circ}$ rotation around point $A$, which maps point $C$ to point $B$. Let point $O$ be mapped to point $D$ during this rotation; then segment $BD$ is the image of segment $CO$; since the length of segments does not change during rotation, $BD = CO = 4$. We obtain quadrilateral $OADB$, in which $OA = AD = 5, BD = 4, OB = 6$, and $\angle OAD = 90^{\circ}$ (see the diagram). We can reason in several ways from here.

![](https://cdn.mathpix.com/cropped/2024_05_06_c901cfc70ca56cbce9b9g-08.jpg?height=492&width=397&top_left_y=208&top_left_x=1572)

First method. Consider a coordinate system where point $O$ has coordinates $(0,0)$, point $A$ has coordinates $(5,0)$, and point $D$ has coordinates $(5,-5)$. We will find the coordinates of point $B(x, y)$, considering that $OB = 6$ and $DB = 4$, i.e.,

$$
\left\{\begin{array}{l}
x^{2}+y^{2}=36 \\
(x-5)^{2}+(y+5)^{2}=16
\end{array}\right.
$$

Subtracting the second equation from the first, we get $10x - 10y - 50 = 20$, hence $x - y = 7$. Substituting $x = y + 7$ into the first equation, we get $2y^2 + 14y + 13 = 0$, from which $y = \frac{-7 \pm \sqrt{23}}{2}, x = \frac{7 \pm \sqrt{23}}{2}$. Since point $B$ must lie on the same side of $AD$ as point $O$, we have $y = \frac{-7 - \sqrt{23}}{2}, x = \frac{7 - \sqrt{23}}{2}$.

Finally, $S = \frac{AB^2}{2} = \frac{(x-5)^2 + y^2}{2} = \frac{((7 - \sqrt{23}) / 2 - 5)^2 + ((7 + \sqrt{23}) / 2)^2}{2} = 13 + \frac{5}{2} \sqrt{23}$.

Second method. First, note that $OD = 5\sqrt{2}, \angle ODA = 45^{\circ}$. Let $\angle ODB = \varphi$. Then, by the cosine rule for triangle $ODB$, we have

$$
\cos \varphi = \frac{OD^2 + BD^2 - OB^2}{2 \cdot OD \cdot BD} = \frac{(5\sqrt{2})^2 + 4^2 - 6^2}{2 \cdot 5\sqrt{2} \cdot 4} = \frac{30}{2 \cdot 5\sqrt{2} \cdot 4} = \frac{3\sqrt{2}}{8}
$$

and then $\sin \varphi = \sqrt{1 - \frac{18}{64}} = \frac{\sqrt{46}}{8}$. Now, by the cosine rule for triangle $ADB$, we get

$$
S = \frac{AB^2}{2} = \frac{AD^2 + BD^2 - 2 \cdot AD \cdot BD \cdot \cos(\varphi + 45^{\circ})}{2} = \frac{25}{2} + 8 - 5 \cdot 4 \cdot \frac{\cos \varphi - \sin \varphi}{\sqrt{2}} = 13 + \frac{5}{2} \sqrt{23}
$$

Second solution. Let $AB = AC = x, \angle OAB = \varphi$. By the cosine rule for triangles $OAB$ and $OAC$, we have:

$$
\begin{aligned}
& 6^2 = x^2 + 5^2 - 10x \cos \varphi \\
& 4^2 = x^2 + 5^2 - 10x \sin \varphi
\end{aligned}
$$

from which

$$
\begin{aligned}
& 10x \cos \varphi = x^2 - 11 \\
& 10x \sin \varphi = x^2 + 9
\end{aligned}
$$

![](https://cdn.mathpix.com/cropped/2024_05_06_c901cfc70ca56cbce9b9g-08.jpg?height=428&width=434&top_left_y=1962&top_left_x=1531)

Squaring these inequalities and adding them, we get a quadratic equation in $x^2$: $100x^2 = 2x^4 - 4x^2 + 202, x^4 - 52x^2 + 101 = 0$. The roots of this equation are

$$
x_{1,2}^2 = 26 \pm 5\sqrt{23}
$$

Note that $26 - 5\sqrt{23} < 25$, and in this case $x < AO$, meaning point $O$ would not lie inside the triangle, so $S = \frac{x^2}{2} = 13 + \frac{5}{2} \sqrt{23}$.

Plan for the third solution. Reflect point $O$ symmetrically with respect to the sides $AB, AC$, and $BC$ of triangle $ABC$; denote the images by $X, Y$, and $Z$ respectively. (see the diagram) Then $AY = AX = AO, CY = CZ = CO, BX = BZ = BO$. By a simple calculation of angles, we see that $\angle YCZ = 2\angle ACB = 90^{\circ}, \angle XBZ = 2\angle ABC = 90^{\circ}, \angle XAY = 2\angle BAC = 180^{\circ}$.

The area of pentagon $XYCZB$ is twice the area of triangle $ABC$. On the other hand, the area of $XYCZB$ is composed of the areas of two right triangles $YCZ$ and $XBZ$, in which we know one leg, and triangle $XYZ$, for which we know all sides, so we can find its area, for example, using Heron's formula.

![](https://cdn.mathpix.com/cropped/2024_05_06_c901cfc70ca56cbce9b9g-09.jpg?height=739&width=719&top_left_y=190&top_left_x=1248)