Find the area of the square with vertex at $C$ The figure below shows $6$ squares connected edge by edge. The top three are unit squares.If $$ar(\Delta ABD)=\frac{3}{2} \times ar(\Delta ABC)$$ Find the area of the square with vertex at $C$.

My try:
I actually attacked this using Analytical geometry:
I assumed the following:
$FB$ is along $X$ axis and $EH$ is along $Y$ axis.
Let $F(0,0),A(1,1),B(2,0),G(0,-a),H(0,-b)$.
Thus the side length of the square $CGHK$ is $b-a$.
Let the side length of the square $LKMD$ is $x$.
So the coordinates of $L$ is $L(b-a-x,-b)$
So the vertex $D$ is $D(b-a-x,x-b)$.
Also the vertex $C$ is $C(b-a,-a)$
Now using the formula for area of triangle using determinant as:
$$\Delta=\frac{1}{2}\left|\begin{array}{lll}
x_{1} & y_{1} & 1 \\
x_{2} & y_{2} & 1 \\
x_{3} & y_{3} & 1
\end{array}\right|$$
We have $$ar(ABD)=\frac{1}{2}\left|\begin{array}{ccc}
b-a-x & x-b & 1 \\
1 & 1 & 1 \\
2 & 0 & 1
\end{array}\right|$$ and
$$ar(ABC)=\frac{1}{2}\left|\begin{array}{ccc}
b-a & -a & 1 \\
1 & 1 & 1 \\
2 & 0 & 1
\end{array}\right|$$
Using the fact that:
$$ar(\Delta ABD)=\frac{3}{2} \times ar(\Delta ABC)$$
We get
\begin{array}{l}
\frac{a+2}{2}=\frac{3}{2}\left(\frac{b-2 a-2}{2}\right) \\
\Rightarrow \quad 6 a-3 b+10=0 \tag{1}
\end{array}
I need help to find another equation so that we can solve the simultaneous equations to find $a,b$ and hence $b-a$.

On your solution, you have some mistakes in the sign. First note that $|b| = 3$. With $F$ as origin, and $|FG| = a$ and $|DM| = x$,
Coordinates of $C$ is $(-3 + a, -a)$ and of $D$ is $(-3 + a - x, -3 + x)$. Now using determinant for area and solving, you get $ \displaystyle a = \frac{1}{2} \ $ so side length of the square with vertex at $C$ is $\displaystyle \frac{5}{2}$.
Here is a solution using trigonometry -
Assume the side length of the square with vertex at $C$ is $x$ and with vertex at $D$ is $y$.
As $\triangle ABC$ and $\triangle ABD$ have the same base $AB$ and given the ratio of area of triangles,
$\displaystyle h_2 = \frac{3 h_1}{2} \ $ where $h_1$ is altitude from $C$ to base $AB$ and $h_2$ is altitidue from $D$ to base $AB$.
If $\angle CBF = \alpha, \angle DBF = \beta$,
$\displaystyle \sin \alpha = \frac{3-x}{BC}, \cos \alpha = \frac{2+x}{BC}$
$\displaystyle \sin \beta = \frac{3-y}{BD}, \cos \beta = \frac{2+x+y}{BD}$
Also note that $ \displaystyle \angle ABF = \frac{\pi}{4}$
$\displaystyle h_1 = BC \sin \big(\frac{\pi}{4} + \alpha) = \frac{5}{\sqrt2}$
$\displaystyle h_2 = BD \sin \big(\frac{\pi}{4} + \beta) = \frac{5+x}{\sqrt2}$
As $ \displaystyle h_2 = \frac{3 h_1}{2}, x = \frac{5}{2}$
So area of square with vertex at $C$ is $\displaystyle \frac{25}{4}$.
