K-Means results interpretation when we have no idea of the number of clusters I have a dataset with 11 variables and 80 000 observations.
I know 2 techniques to find evidence of clusters in a dataset: hierarchical clustering and k-means. I can't use the hierarchical clustering because the dataset is too large so I try the k-means approach.
With both my handmade function and the sklearn KMeans function, I get 4 clusters (I used the kneed.KneeLocator function to do so). My question in not about the technique, but more about whether this makes sense:
I find 4 clusters, great! But is that an evidence of the presence of clusters in the dataset? Wouldn't k-means have found an ideal number of clusters even if this was not clear?


$k$-means, as all the other clustering algorithms, are unsupervised methods. What this means is that they don’t use labels. It’s not only about not needing the labels, but you don’t need to assume that any such labels exist. The algorithms just group “similar” things together, using different definitions of what that means. There may be multiple different but equally good clustering solutions for the dataset. In many cases the choice between the solutions would be rather arbitrary. So you don’t need to know the number of clusters, often it is rather how many clusters you choose to use.
From the How to decide on the correct number of clusters? thread you can learn about tools that could help you with making the decision.
