CLT may fail under this condition? CLT says that when sample size n goes to infinity, the sample sum or average curve converges to normal distribution.
My question is: if the sample size goes closer or even equal to the population size, wouldn't the average values cram so closely near the mean, that violate the "bell shape" distribution?

No, it does not, since Gaussian distribution converges to Dirac Delta for very small s.d. $\epsilon$, $\delta(x)=\lim\limits_{\epsilon\to 0^+}\frac{e^{-\frac{1}{2}\frac{x^2}{\epsilon^2}}}{\epsilon\sqrt{2\pi}}$.
By CLT, for i.i.d. r.v.s $X_1,\ldots, X_n$, with population mean $\mu$ and finite variance $\sigma^2$,
$\bar{X_n}-\mu \overset{D}{\to} \mathcal{N}(0,\sigma^2/n)$. When we have large sample size  $n$, we have $\epsilon=\dfrac{\sigma}{\sqrt{n}} \to 0$, so the shape of the distribution approaches Dirac Delta, which is a unit impulse function, refer to this: https://en.m.wikipedia.org/wiki/Dirac_delta_function.
For example, you can use the following R code to visualize the shape of the pdf of a standard normal r.v., with variance $\dfrac{\sigma^2}{n}$, for some value of population s.d. $\sigma$ (e.g., 10) different sample size $n$, as shown in the next animation (approaches Dirac Delta with large $n$).
 σ <- 10
 plot(x, dnorm(x, sd = sigma/sqrt(n)), ylab='pdf', type='l', 
         main=paste('normal pdf with variance σ^2/n, σ = 10, n =', n))


