Area of equilateral triangle inscribed in right triangle An Isosceles right triangle  $ABC$  , $AB=BC= 4 cm$
Point $p$ is a midpoint of $BC$ , points $q$, $s$  lies on $AC$,$AB$ respectively  , such that the triangle $pqs$ is  an equilateral triangle ; 
what is the area of triangle $pqs$ 


Based on your description, let's define:
$$ \begin{align}
\mbox{Hypotenuse AC}:\; y &= -x +4 \\
p &= (2,0) \\
q &= (x, y=-x+4) \\
s & = (0,h)
\end{align}$$
The three sides are equal, hence:
$$ \begin{align}
\mbox{sp = pq} \Rightarrow 
4+h^2 &= (x-2)^2 + (x-4)^2 \quad (1)\\
\mbox{sp = sq} \Rightarrow 
4+h^2 &= x^2 + (h+x-4)^2, \quad(2) \\
\mbox{pq = sq} \Rightarrow 
(x-2)^2 + (x-4)^2 &=  x^2 + (h+x-4)^2. \quad(3)\\
\end{align}$$
There are 3 equations with only 2 variables $x$ and $h$ but only two equations are needed to solve for the variables. Let's say we solve $(2)$ and $(3)$ for $x$ and $h$. Then we plug the solution in to $(1)$ and if $(1)$ is not satisfied, then the problem has no solution. 
From $(3)$:
$$ x = \frac{4 + 8h -h^2}{4 + 2h}. \quad(4) $$
Plugging $(4)$ into $(2)$ will result in an equation in $h$ alone. This equation can be solved numerically. Once $h$ is obtained, $x$ will be found by $(4)$.
The next step is to plug the results for $x$ and $h$ into $(1)$. If the equality fails, the problem has no solution.
If the problem has a solution, then the area of the triangle $spq$ is given by
$$ A = \frac{\sqrt3}{4}(sp)^2 =  \frac{\sqrt3}{4}(4+h^2). $$
