Consider these two operations:
\begin{align*} a \, \blacklozenge \, b &= a^2 - b^2\\ a \, \bigstar \, b &= (a - b)^2 \end{align*}
What is the output of $(5 \, \blacklozenge \, 3) \, \bigstar \, 6?$
$\textbf{(A) } {-}20 \qquad \textbf{(B) } 4 \qquad \textbf{(C) } 16 \qquad \textbf{(D) } 100 \qquad \textbf{(E) } 220$
We can substitute $5$, $3$, and $6$ into the functions' definitions:
\begin{align*} (5 \, \blacklozenge \, 3) \, \bigstar \, 6 &= \left(5^2-3^2\right) \, \bigstar \, 6 \\ (5 \, \blacklozenge \, 3) \, \bigstar \, 6 &= \left(25-9\right) \, \bigstar \, 6 \\ &= 16 \, \bigstar \, 6 \\ &= (16-6)^2 \\ &= \boxed{\textbf{(D) } 100}. \end{align*}
~pog ~MathFun1000 (Minor Edits)