CLT may fail under this condition? CLT says that when sample size n goes to infinity, the sample sum or average curve converges to normal distribution.
My question is: if the sample size goes closer or even equal to the population size, wouldn't the average values cram so closely near the mean, that violate the "bell shape" distribution?

The central limit theorem applies to infinite sequences of random variables $X_1,X_2,X_3,...$ rather than finite vectors of random variables.  This is evident in the fact that we take the limit $n \rightarrow \infty$ in these theorems.  So what this means is that these problems deal implicitly with an infinite population (or "superpopulation" if you prefer).  When we take this kind of limit, the sample size never goes close to or equal to the population size, since the latter is infinite.  Regardless of how big $n$ gets in the limiting analysis, it is still finite and so it is never close to the population size.
Now, it is of course possible to look at the behaviour of the sample mean for a finite population of size $N \in \mathbb{N}$.  In this case, if you take the sample size $n$ up to the population size $N$ then the sample mean will equal the population mean exactly.  If your analysis conditions on knowledge of the population mean (or treats this as a fixed constant) then the distribution of the sample mean is a point-mass distribution on the population mean value.  Similarly, if the sample mean is close to the (finite) population mean, then its distribution also will not be bell shaped.  (You can think of this like the CLT in reverse; if the number of unsampled datapoints is small then the distribution of the unsampled mean is not well approximated by the normal, and the sample mean is an affine function of this value, so it is also not well approximated by the normal.)
