Find the coefficient of $x^n$ in the generating functions: $g(x) = \frac{x^3}{(1+x)^5 (1−x)^6}$ One of the problems in my Discrete Math course states that we need to find the coefficient of $x^n$ in generating function $g(x) = \frac{x^3} {(1+x)^5 (1−x)^6}$
I separated $g(x) = \frac{x^3} {(1+x)^5 (1−x)^6}$ into $x^3$ * $\frac{1}{(1+x)^5}$ * $\frac{1}{(1-x)^6}$
Then got $f(x) = x^3 * \sum_{k=0}^\infty \binom{n+4}{4}(-1)^kx^k * \sum_{k=0}^\infty \binom{n+5}{5}x^k$
I dont know what to do from here, can you tell me what is the next step?

The key is the generalized binomial theorem
in the handy special case
$(1-x)^{-s}
=\sum_{k=0}^{\infty} \binom{s+k-1}{k}x^k
=\sum_{k=0}^{\infty} \binom{s+k-1}{s-1}x^k
$.
This implies that
$(1+x)^{-s}
=\sum_{k=0}^{\infty} \binom{s+k-1}{k}(-1)^kx^k
=\sum_{k=0}^{\infty} \binom{s+k-1}{s-1}(-1)^kx^k
$
and,
for any $m$,
$(1-x^m)^{-s}
=\sum_{k=0}^{\infty} \binom{s+k-1}{k}x^{km}
=\sum_{k=0}^{\infty} \binom{s+k-1}{s-1}x^{km}
$.
For $m=2$ this is
$(1-x^2)^{-s}
=\sum_{k=0}^{\infty} \binom{s+k-1}{k}x^{2k}
=\sum_{k=0}^{\infty} \binom{s+k-1}{s-1}x^{2k}
$.
Your problem is
$\begin{array}\\
g(x) 
&= \dfrac{x^3} {(1+x)^5 (1−x)^6}\\
&= \dfrac{x^3(1+x)} {(1+x)^6 (1−x)^6}\\
&= \dfrac{x^3(1+x)} {(1−x^2)^6}\\
&= \dfrac{x^3} {(1−x^2)^6}+\dfrac{x^4} {(1−x^2)^6}\\
\end{array}
$
The results above
should make this
straightforward.

Actually,
it's not that simple.
Here's the rest.
$\begin{array}\\
\dfrac{x^3} {(1−x^2)^6}
&=x^3\sum_{k=0}^{\infty} \binom{k+5}{5}x^{2k}\\
&=\sum_{k=0}^{\infty} \binom{k+5}{5}x^{2k+3}\\
&=\sum_{k=0}^{\infty} \binom{k+5}{5}x^{2(k+1)+1}\\
&=\sum_{k=1}^{\infty} \binom{k+4}{5}x^{2k+1}\\
\\
\dfrac{x^4} {(1−x^2)^6}
&=x^4\sum_{k=0}^{\infty} \binom{k+5}{5}x^{2k}\\
&=\sum_{k=0}^{\infty} \binom{k+5}{5}x^{2k+4}\\
&=\sum_{k=0}^{\infty} \binom{k+5}{5}x^{2(k+2)}\\
&=\sum_{k=2}^{\infty} \binom{k+3}{5}x^{2k}\\
\end{array}
$
This gives the coefficients
of the even and odd terms.
To map these cases
into a single one,
use
$h(k)
=k-2\lfloor k/2 \rfloor
$
where
$h(k) = 0$ for even $k$
and
$h(k) = 1$ for odd $k$.
Since we want
even $k \to \lfloor k/2 \rfloor+3$
and
odd $k \to \lfloor k/2 \rfloor+4$,
we want
$\begin{array}\\
k 
&\to \lfloor k/2 \rfloor+3+h(k)\\
&\to \lfloor k/2 \rfloor+3+k-2\lfloor k/2 \rfloor\\
&\to k+3-\lfloor k/2 \rfloor\\
\end{array}
$
Therefore
$g(x)
=\sum_{k=3}^{\infty} x^k\binom{k+3-\lfloor k/2 \rfloor}{5}
$.
