If $a,b$, and $d$ are the lengths of a side, a shortest diagonal and a longest diagonal, respectively, 
of a regular nonagon (see adjoining figure), then
$\textbf{(A) }d=a+b\qquad \textbf{(B) }d^2=a^2+b^2\qquad \textbf{(C) }d^2=a^2+ab+b^2\qquad\\ \textbf{(D) }b=\frac{a+d}{2}\qquad  \textbf{(E) }b^2=ad$
By the law of cosines we can get the following expressions for $d^{2}$ and $b^{2}$:
\[d^{2}=2b^{2}(1-\cos (100^\circ))\] \[b^{2}=2a^{2}(1-\cos (140^\circ))\]
We can substitute what we got for $b^2$ into the expression for $d^{2}$:
\[d^{2}=4a^{2}(1-\cos (140^\circ))(1-\cos (100^\circ))=4a^{2}(1-(\cos (100^\circ)+\cos (140^\circ))+\cos (100^\circ)\cos (140^\circ))\]
Now apply sum-to-product and product-to-sum identities:
\[d^{2}=4a^{2}(1-(2\cos (120^\circ)\cos (120^\circ))+\frac{1}{2}(\cos (240^\circ)+\cos (40^\circ)))\]
Simplifying further gives us: \[d^{2}=4a^{2}(1+\cos (20^\circ)-\frac{1}{4}+\frac{1}{2}\cos (40^\circ))\]
After using the fact that $\cos (40^\circ)=2\cos^2 (20^\circ)-1$, it's not hard to see that the expression in the parentheses is equal to $(\cos (20^\circ)+\frac{1}{2})^{2}$. So we can square-root both sides to find the expression for $d$:
\[d=2a(\cos (20^\circ)+\frac{1}{2})\]
Now let's look at the expression for $b^2$. We can apply the reverse of the double angle identity to show that $1-\cos (140^\circ)$ equals $2\sin^2 (70^\circ))$. So if we square root the entire expression we get that \[b=2a\sin (70^\circ)=2a\cos (20^\circ)\]
We now have everything in terms of $a$. Luckily when we consider choice $\textbf{A}$ we can verify without much work that this must be the answer.
Solution by harita19

NOTE: a much easier solution exists by drawing some lines and recognizing that the nonagon is cyclic, but for those of use who use algebra in every geometry problem, this is the best solution.