What is the coefficient of the term $x^3y^5$, as a result of the binomial expansion of the following term? We have the term
$(1+xy+y^2)^n$ 
If we expand it using the binomial theorem, why is the factor of the term $x^3y^5$ the following: $4{n\choose 4}$? (The binomial coefficient n choose 4 multiplied by 4)

It is convenient to use the coefficient of operator $[x^n]$ to denote the coefficient of $x^n$ in a series.

We obtain
  \begin{align*}
\color{blue}{[x^3y^5]}&\color{blue}{(1+xy+y^2)}\\
&=[x^3y^5]\sum_{j=0}^n\binom{n}{j}(xy)^j(1+y^2)^{n-j}\tag{1}\\
&=[y^5]\binom{n}{3}y^3(1+y^2)^{n-3}\tag{2}\\
&=\binom{n}{3}[y^2]\sum_{j=0}^{n-3}\binom{n-3}{j}y^{2j}\tag{3}\\
&=\binom{n}{3}\binom{n-3}{1}\tag{4}\\
&=\frac{n(n-1)(n-2)}{3!}\cdot(n-3)\\
&=4\cdot\frac{n(n-1)(n-2)(n-3)}{4!}\\
&\,\,\color{blue}{=4\binom{n}{4}}
\end{align*}

Comment:


*

*In (1) we apply the binomial theorem to $(xy+(1+y^2))^n$. This way we can easily select the coefficient of $x^3$.

*In (2) we select the coefficient of $x^3$.

*In (3) we use the formula $[y^{p-q}]A(y)=[y^p]y^qA(y)$ and apply the binomial theorem again.

*In (4) we select the coefficient of $y^2$.
