Find number of ways to choose $3n$-subset with repetitions from set $\left\{A,B,C\right\}$ Find number of ways to choose $3n$-subset with repetitions from set $\left\{A,B,C\right\}$ such that: 
1. Letter $A$ occur at most $2n$
2. Letter $B$ occur at most $2n$
3. Letter $C$ occur odd times
Approach
I want to use there enumerators. Ok, so a factor responsible for $A$ will be
$$(1+x+x^2+ \cdots + x^{2n}) $$
(We can choose $A$ $0$ times, $1$ time, ... $2n$ times). The same will be for $B$.  
Enumerator for $C$ will be
$$(x+x^3+x^5 + \cdots) $$
(We can choose $C$ 1 time, 3 times, etc)

Ok, so I want to find 
$$[x^{3n}](1+x+x^2+ \cdots + x^{2n})(1+x+x^2+ \cdots + x^{2n})(x+x^3+x^5 + \cdots) = $$
$$ [x^{3n}] \left(\frac{1-x^{2n+1}}{1-x}\right)^2 \cdot\frac{x}{1-x^2} $$
but... how I can get from there factor at $x^{3n}$?


We obtain for $n\geq  1$:
\begin{align*}
[x^{3n}]&\left(\frac{1-x^{2n+1}}{1-x}\right)^2\frac{x}{1-x^2}\tag{1}\\
&=[x^{3n-1}]\frac{1-2x^{2n+1}}{(1-x)^2\left(1-x^2\right)}\tag{2}\\
&=\left([x^{3n-1}]-2[x^{n-2}]\right)\sum_{k=0}^\infty\binom{-2}{k}(-x)^k\sum_{j=0}^\infty x^{2j}\tag{3}\\
\end{align*}

Comment:

*

*In  (2) we apply the rule $[x^{p-q}]A(x)=[x^p]x^qA(x)$ and we expand the numerator skipping the term  $x^{4n+2}$ which  does not contribute to $[x^{3n-1}]$.


*In (3) we  apply the rule from (2) again and do a geometric and a binomial series expansion.

Next we calculate the coefficient of $x^n$. We obtain from (3)
\begin{align*}
[x^n]&\sum_{j=0}^\infty x^{2j}\sum_{k=0}^\infty\binom{k+1}{1}x^k\tag{4}\\
&=\sum_{j=0}^{\left\lfloor n/2\right\rfloor}[x^{n-2j}]\sum_{k=0}^\infty (k+1)x^k\tag{5}\\
&=\sum_{j=0}^{\left\lfloor n/2\right\rfloor}(n-2j+1)\tag{6}\\
&=(n+1)\sum_{j=0}^{\left\lfloor n/2\right\rfloor}1-2\sum_{j=0}^{\left\lfloor n/2\right\rfloor}j\\
&=(n+1)\left(\left\lfloor\frac{n}{2}+1\right\rfloor\right)-\frac{n}{2}\left(\left\lfloor\frac{n}{2}+1\right\rfloor\right)\\
&=\begin{cases}
(n+1)\left(\frac{n}{2}+1\right)-\frac{n}{2}\left(\frac{n}{2}+1\right)&\qquad\qquad\qquad n\text{ even}\\
(n+1)\left(\frac{n-1}{2}+1\right)-\frac{n-1}{2}\left(\frac{n-1}{2}+1\right)&\qquad\qquad\qquad n\text{ odd}\\
\end{cases}\\
&=\begin{cases}
\frac{1}{4}(n+2)^2&\qquad\qquad\qquad\qquad\qquad\qquad\qquad\quad\  n\text{ even}\\
\frac{1}{4}(n+2)^2-\frac{1}{4}&\qquad\qquad\qquad\qquad\qquad\qquad\qquad\quad\ n\text{ odd}\tag{7}\\
\end{cases}
\end{align*}

Comment:

*

*In (4) we use the binomial identity $\binom{-p}{q}=\binom{p+q-1}{p-1}(-1)^q$.


*In (5) we apply again $[x^{p-q}]A(x)=[x^p]x^qA(x)$ and we set the upper limit of the outer sum to $\left\lfloor\frac{n}{2}\right\rfloor$ since the coefficient is non-negative.


*In (6) we select the coefficient of $x^{n-2j}$.
We can now evaluate (3) with the help of (7) and note that if $n$ is even we have odd $3n-1$ and even $n-2$. On the other hand if $n$ is odd we have even $3n-1$ and odd $n-2$.

We obtain from (3) and (7)
\begin{align*}
\color{blue}{[x^{3n}]}&\color{blue}{\left(\frac{1-x^{2n+1}}{1-x}\right)^2\frac{x}{1-x^2}}\\
&=\left([x^{3n-1}]-2[x^{n-2}]\right)\sum_{k=0}^\infty\binom{-2}{k}(-x)^k\sum_{j=0}^\infty x^{2j}\\
&=\begin{cases}
\frac{1}{4}(3n+1)^2-\frac{1}{4}-2\cdot\frac{1}{4}n^2&\qquad\qquad\qquad n\text{ even}\\
\frac{1}{4}(3n+1)^2-2\left(\frac{1}{4}n^2+\frac{1}{4}\right)&\qquad\qquad\qquad n\text{ odd}\\
\end{cases}\\
&\,\,\color{blue}{=\frac{1}{4}\left(7n^2+6n+3[[n\text{ odd}]]\right)}\tag{8}
\end{align*}

In (8) we use Iverson brackets as compact notation for even and odd cases.
