Why is the importance of variable used in clustering proportional to its variance? Equation 14.27 ESL Basically, I don't understand how the variance was derived here:
$$
\overline{d}_j = \frac{1}{N^2}\sum_{i=1}^N\sum_{i^{'}=1}^N(x_{ij}-x_{i^{'}j})^2 = 2 \cdot var_j
$$
That is an equation 14.27 from Elements of Statistical Learning.
Excerpt from section 14.3.3:

The influence of the jth attribute Xj on object dissimilarity
D(xi , xi′ ) depends upon its relative contribution to the average
object dissimilarity measure over all pairs of observations in the data set
$$
\overline{D} = \frac{1}{N^2}\sum_{i=1}^N\sum_{i^{'}=1}^N D(x_{i},x_{i^{'}}) = \sum_{i^{'}=1}^N w_j \cdot \overline{d_j}
$$
with
$$ 
\overline{d}_j = \frac{1}{N^2}\sum_{i=1}^N\sum_{i^{'}=1}^N d(x_{ij},x_{i^{'}j})
$$

Using squared-error distance (d(xij, xi'j)) for each coordinate, equation 14.27 can be found.

Easing up the notation, let's say we have a set of values ($N$ values) in an array. The following quantity
$$\begin{align}\frac{1}{N^2}\sum_i \sum_j (x_i-x_j)^2&=\frac{1}{N^2}\sum_i\left(\sum_j x_i^2-2x_ix_j+x_j^2\right)\\&=\frac{1}{N} \sum_i x_i^2 -\frac{2}{N^2}\sum_i x_i\sum_j x_j+\frac{1}{N}\sum_j x_j^2\\&=\frac{2}{N}\sum_i x_i^2 - 2\bar{x^2}=2\overline {x^2}-2\bar x^2 = \widehat{2\operatorname{var}(x)}\end{align}$$
I have ignored $j$ in your notation as it doesn't contribute to anything, and changed $i'$ to $j$ instead to make notation more clear. I've also ignored the upper and lower limits of the summations, as both are from $1$ to $N$.
The expression is two times the estimate of population variance because $$\operatorname{var}(X)=E[X^2]-E[X]^2$$
