Answers.Body: <p>You can use K-means even without knowing a priori the amount of classes. Take a look at the definition of <a href="https://en.wikipedia.org/wiki/Silhouette_(clustering)" rel="nofollow noreferrer">Silhouette score</a>, it's a generic approach applicable to any clustering method that requires an input value for the final amount of clusters to generate.</p>
  <p>The silhouette score represents an average between the inner similarity of each cluster elements and outer dissimilarity between each cluster element and all elements belonging to different clusters. The higher the silhouette score, the better the clusters split the data.</p>
  <p>Only drop down of the approach is the necessity to run the clustering algorithm with several different initialization values for n: number of clusters. That's the only way to get a graph like the one show below, in other to understand the optimal value for n (in this fake example 3).</p>
  <p><a href="https://i.sstatic.net/iAWnF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iAWnF.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-18T00:07:16.143Z
Answers.LastActivityDate: 2022-01-18T00:07:16.143Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>Recently I trained a BYOL model on a set of images to learn an embedding space where similar vectors are close by. The performance was fantastic when I performed approximate K-nearest neighbours search.</p>
  <p>Now the next task, where I am facing a problem is to find a clustering algorithm that uncovers a set of clusters using the embedding vectors generated by the BYOL trained feature extractor [dimension of the vector is 1024 &amp; there are 1 million vectors]. I have no information apriori about the number of classes i.e. clusters in my dataset &amp; thus cannot use Kmeans. Is there any scalable clustering algorithm that can help me uncover such clusters. I tried to use FISHDBC but the repository does not have good documentation.</p>
  
Tags: <deep-learning><clustering><embeddings><self-supervised-learning><dimensionality-reduction>
Title: Perform clustering on high dimensional data
UserDisplayName: VEDANT JOSHI
ThreadText: # Perform clustering on high dimensional data
  
  (Asked by: username_0 on 2022-01-14)
  
  Recently I trained a BYOL model on a set of images to learn an embedding space where similar vectors are close by. The performance was fantastic when I performed approximate K-nearest neighbours search.
  
  Now the next task, where I am facing a problem is to find a clustering algorithm that uncovers a set of clusters using the embedding vectors generated by the BYOL trained feature extractor [dimension of the vector is 1024 & there are 1 million vectors]. I have no information apriori about the number of classes i.e. clusters in my dataset & thus cannot use Kmeans. Is there any scalable clustering algorithm that can help me uncover such clusters. I tried to use FISHDBC but the repository does not have good documentation.
  
  **Tags:** deep-learning, clustering, embeddings, self-supervised-learning, dimensionality-reduction
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-18. Score: 1
  
  You can use K-means even without knowing a priori the amount of classes. Take a look at the definition of [Silhouette score](https://en.wikipedia.org/wiki/Silhouette_(clustering)), it's a generic approach applicable to any clustering method that requires an input value for the final amount of clusters to generate.
  
  The silhouette score represents an average between the inner similarity of each cluster elements and outer dissimilarity between each cluster element and all elements belonging to different clusters. The higher the silhouette score, the better the clusters split the data.
  
  Only drop down of the approach is the necessity to run the clustering algorithm with several different initialization values for n: number of clusters. That's the only way to get a graph like the one show below, in other to understand the optimal value for n (in this fake example 3).
  
  [![enter image description here](https://i.sstatic.net/iAWnF.png)](https://i.sstatic.net/iAWnF.png)
  
  
  ---
  
  