Find the coefficient of $x^n$ in the generating functions: $g(x) = \frac{x^3}{(1+x)^5 (1−x)^6}$ One of the problems in my Discrete Math course states that we need to find the coefficient of $x^n$ in generating function $g(x) = \frac{x^3} {(1+x)^5 (1−x)^6}$
I separated $g(x) = \frac{x^3} {(1+x)^5 (1−x)^6}$ into $x^3$ * $\frac{1}{(1+x)^5}$ * $\frac{1}{(1-x)^6}$
Then got $f(x) = x^3 * \sum_{k=0}^\infty \binom{n+4}{4}(-1)^kx^k * \sum_{k=0}^\infty \binom{n+5}{5}x^k$
I dont know what to do from here, can you tell me what is the next step?

Here is a variation using the coefficient of operator $[x^n]$ which denotes the coefficient of $x^n$ of a series.

We start with $f(x)$ and obtain for $n\geq 3$:
\begin{align*}
\color{blue}{[x^n]f(x)}&=[x^n]x^3\sum_{k=0}^\infty\binom{k+4}{4}(-1)^kx^k\sum_{l=0}^\infty\binom{l+5}{5}x^l\tag{1}\\
&=[x^{n-3}]\sum_{k=0}^\infty\binom{k+4}{4}(-1)^kx^k\sum_{l=0}^\infty\binom{l+5}{5}x^l\tag{2}\\
&=\sum_{k=0}^{n-3}\binom{k+4}{4}(-1)^k[x^{n-3-k}]\sum_{l=0}^\infty\binom{l+5}{5}x^l\tag{3}\\
&=\sum_{k=0}^{n-3}\binom{k+4}{4}(-1)^k\sum_{l=0}^\infty\binom{l+5}{5}[x^{n-3-k}]x^l\\
&\,\,\color{blue}{=\sum_{k=0}^{n-3}\binom{k+4}{4}\binom{n-k+2}{5}(-1)^k}\tag{4}
\end{align*}

Comment:

*

*In (1) we write the sums using different indices $k$ and $l$ which helps to not mix them up.


*In (2) we absorb $x^3$ by applying the rule $[x^{p-q}]A(x)=[x^p]x^qA(x)$.


*In (3) we use the linearity of the coefficient of operator and apply the rule $[x^{p-q}]A(x)=[x^p]x^qA(x)$ again. Note the upper index is set to $n-3$, since other indices do not contribute to the coefficient of $x^{n-3}$.


*In (4) we select the coefficient of $x^{n-3-k}$ which implies choosing $l=n-3-k$. Here we use $[x^p]x^q=\begin{cases}1&p=q\\0&p\ne q\end{cases}$.
