How to look into Agglomerative linkage matrix to find optimal number of clusters using python I am generating the linkage matrix on 73k data-points. The objective is to find the cut-off point by visualising the distance value from linkage matrix.
The code is:
import scipy.cluster.hierarchy as sch
from sklearn.cluster import AgglomerativeClustering

Z = sch.linkage(u, 'complete',metric='cosine') #u is 73k*768 dimension embedding vectors.

Output:
Z[0]
array([ 52.     ,  53.     ,   0.04151,   2.     ])
We can see that each row of the resulting array has the format [idx1, idx2, dist, sample_count]
Can I use this matrix to identify rough cut-off point/ optimal number of clusters.
I tried to plot line graph of all the distance value from Z and identify any jump or Knee-point, but it looks cluttered. But using this how can I decide optimal clusters?
Is there a pythonic way to use this matrix to come up with rough number of clusters?

One way could be to visualize the linkage matrix as a dentrogram
import scipy.cluster.hierarchy as sch
import matplotlib.pyplot as plt

u = ...
Z = sch.linkage(u, 'complete',metric='cosine')
sch.dendrogram(Z)
plt.show()

An indication for a useful number of clusters can be obtained by finding the largest horizontal space that does not have any vertical lines, i.e., the space with the longest vertical lines. This can be a good indication since then clusters are separated in a 'good way'. However, it strongly depends on the application.
