A cube of edge $3$ cm is cut into $N$ smaller cubes, not all the same size.  If the edge of each of the smaller cubes is a whole number of centimeters, then $N=$
$\text{(A)}\ 4 \qquad \text{(B)}\ 8 \qquad \text{(C)}\ 12 \qquad \text{(D)}\ 16 \qquad \text{(E)}\ 20$
If none of the cubes have edge length $2$, then all of the cubes have edge length $1$, meaning they all are the same size, a contradiction.
It is clearly impossible to split a cube of edge $3$ into two or more cubes of edge $2$ with extra unit cubes, so there is one $2\times 2\times 2$ cube and $3^3-2^3=19$ unit cubes. 
The total number of cubes, $N$, is $1+19=20\rightarrow \boxed{\text{E}}$.