Why spectral clustering results in disjointed cluster?
I'm working on a project where I have to dynamically cluster the position of objects with respect to one coordinate. So I'm essentially dealing with subsequent frames and each frame represents a one-dimensional dataset. The intuition behind clustering is to form clusters out of points that are in similar distance to other points within the cluster and can be naturally connected. I use spectral clustering due to its ability to cluster points by their connectedness and not the absolute location and I set rbf kernel due to its non-linear transofrmation of distance. However, in some frames the algorithm results in unnatural assignments. One example is
import numpy as np
from sklearn.cluster import SpectralClustering
X = np.array([[51.08354988], [57.10594997], <PHONE_NUMBER>], <PHONE_NUMBER>],
<PHONE_NUMBER>], <PHONE_NUMBER>], <PHONE_NUMBER>], <PHONE_NUMBER>], <PHONE_NUMBER>],
<PHONE_NUMBER>]])
clustering = SpectralClustering(n_clusters = 4, random_state = 42,
gamma = 5 / (X.max() - X.min()))
clustering.fit(X)
and the result of clustering is presented in a form of swarm plot below, so only x coordinate matters here (each color represents a cluster and labels are array indices):
What I cannot understand is why points marked as red are clustered together as the similarity between points {4, 7} and {5, 8, 9} should be really low. My first thought was that maybe this is caused by unlucky, random initialisation, but I tried with many different random states and the resulting clusters seems to be persistant. So I guess this is connected to the chosen affinity measure (rbf_kernel) and its gamma parameter. 506
As points move with each frame, and distances between them are kind of dependent on their overall range, I tried to set gamma to 5 / (X.max() - X.min()). The intuition behind this was that if the range is bigger, then the distances between points are usually bigger and we should penalise them more to obtain similar values of exp(-gamma * ||x-y||^2) to those obtained within smaller range. But it doesn't seem to work as expected and results in faulty clustering where red cluster is formed out of points divided by green cluster). My expectation would be rather clusters formed as follows: {0, 1, 4, 7}, {2, 3}, {9, 8, 5} and {6} or {0}, {1, 4, 7}, {2, 3}, {5, 6, 8, 9}.
So my question are:
1. Is affinity choice and its gamma parameter really the problem here?
2. If so, how can I choose gamma better?
3. Otherwise, what approach should I consider to deal with faulty assignments with separated points within the same clusters as presented here?
4. (Side question) Is there any measure/index that would be suitable to automatically compare clusterings with different number of clusters?
@Edit: As it can be observed under this link, those separated clusters occurs for a short period of time, but still, the problem seems to be recurrent. 506