CLT may fail under this condition? CLT says that when sample size n goes to infinity, the sample sum or average curve converges to normal distribution.
My question is: if the sample size goes closer or even equal to the population size, wouldn't the average values cram so closely near the mean, that violate the "bell shape" distribution?

If you're going to run an asymptotic argument with $n$ close to $N$, you need sequences of finite populations and samples.  For each $m=1,2,3,\dots$ suppose you have a population of size $N_m$ and a sample of size $n_m$, with $N_m\geq n_m$ and $n_m\to\infty$.  We'll need some assumptions about the populations; we can go back and work out what was needed after the argument
We know already (under reasonable assumptions) that if $n_m\to\infty$ we have a Central Limit Theorem, either with $n_m/N_n\to 0$ or with $n_m/N_n\to c\in(0,1)$. If $\mu_m$ is the true finite population mean and $\sigma^2$ is the limit of the true population variances then
$$\sqrt{n_m}(\bar X_{n_m}-\mu_m)\stackrel{d}{\to} N(0,\sigma^2)$$
Suppose $n_m$ is really big, so that the number of unsampled individuals does not go to infinity.  In that case the CLT does fail. In the extreme case, suppose $n_m=N_m-1$, so that only one individual is not sampled.  We could try to get a scaled mean to be Normal in a few ways
First, we could try $\sqrt{n_m}(\bar X_{n_m}-\mu_m)$. That doesn't work because there's only one observation's difference between $\bar X_{n_m}$ and $\mu_m$: we get $$\sqrt{n_m}(\bar X_{n_m}-\mu_m)=\sqrt{n_m}\cdot O_p(1/n_m)\stackrel{p}{\to}0$$
We could rescale and try $n_m(\bar X_{n_m}-\mu_m)$. That's of the right order, but it is just equal $X_{n_m}-\mu$, the one unsampled observation centered minus the true mean.  Or we could scale by $\sqrt{N_m-n_m}=1$, the square root of the unsampled size, but that goes to zero. Or scale the total to get
$$\frac{N_m}{\sqrt{N_m-n_m}}(\bar X_{n_m}-\mu_m)$$
which is again just the unsampled observation minus the mean.
If $N_m-n_m$ is bigger than one but bounded, you get the same sort of result: depending on the scaling it blows up, goes to zero, or gives you a finite sum that doesn't converge to Normal.
However, if $N_n-n_m\to\infty$ you are good. The unsampled observations are an infinite sum that then does satisfy a CLT, and
$$\frac{1}{\sqrt{N_m-n_m}}\left(\sum_{i=1}^{n_m} (X_i-\mu_m)\right)\stackrel{d}{\to}N(0,\sigma^2)$$
so
$$\frac{n_m}{\sqrt{N_m-n_m}}\left(\bar X_{n_m}-\mu_m\right)\stackrel{d}{\to}N(0,\sigma^2)$$
It's still true, though, that
$\sqrt{n_m}(\bar X_{n_m}-\mu_m)$
will blow up.
Ok, so what did we need to assume? I think it would suffice that the population variances $$\sigma^2_m=\frac{1}{N_m}\sum_{i=1}^{N_m} (X_i-\mu_m)^2$$
converge to $\sigma^2$, and that the population third absolute moments
$$\kappa_m=\frac{1}{N_m}\sum_{i=1}^{N_m} |X_i-\mu_m|^3$$
are bounded. This could be the assumption, or we could treat the populations as randomly generated and make assumptions about the data generating process that forces these to hold with probability one.
