Question: How to count unlabeled balls and labeled boxes case

each box can have more than one balls or some box may have no balls

use example , 10 labeled boxes, 8 unlabeled balls

welcome to use polynomial counting

better have closed formula or generating function

would like to know Prob(a labelled box have 3 balls) too

Answer: 
First, there small typo in the RHS of the first equation (the rightmost vector should have $a\_{n+1}$ as first component).

Basically, without using the matrix approach, your recurrence relation has characteristic polynomial 
$$
P(X) = X^2-4X+4
$$
whose only root (with multiplicity 2) is 2. Hence, [a general solution of the recurrence](http://en.wikipedia.org/wiki/Recurrence_relation#General_methods) will be of the form $(\alpha n+\beta)2^n$.
There is no need to be that complicated. 
Here is the trick.
an+2=4an+1−4an, then an+2-2an+1=2(an+1-2an).
So we can conclude that {an+1-2an} is an geometric progression.
 (an+1-2an)/(a2-2a1)=2^(n-1),
an+1-2an=-2^(n-1),
2an-4an-1=-2^(n-2)\*2=-2^(n-1),
4an-1-8an-2=-2^(n-3)\*4=-2^(n-1),
.......
2^(n-1)\*a2-2^n\*a1=-2^(n-1),
sum up ,we get an+1-2^n=-n\*2^(n-1),
so an=2^(n-1)-(n-1)\*2^(n-2)=(3/4-n/4)\*2^n.