  Popular clustering algorithms based on usual distance functions (e.g.,
Euclidean distance) often suffer in high dimension, low sample size (HDLSS)
situations, where concentration of pairwise distances has adverse effects on
their performance. In this article, we use a dissimilarity measure based on the
data cloud, called MADD, which takes care of this problem. MADD uses the
distance concentration phenomenon to its advantage, and as a result, clustering
algorithms based on MADD usually perform better for high dimensional data.
Using theoretical and numerical results, we amply demonstrate it in this
article.
  We also address the problem of estimating the number of clusters. This is a
very challenging problem in cluster analysis, and several algorithms have been
proposed for it. We show that many of these existing algorithms have superior
performance in high dimensions when MADD is used instead of the Euclidean
distance. We also construct a new estimator based on penalized Dunn index and
prove its consistency in the HDLSS asymptotic regime, where the sample size
remains fixed and the dimension grows to infinity. Several simulated and real
data sets are analyzed to demonstrate the importance of MADD for cluster
analysis of high dimensional data.

Methodology