title: Why is k-means clustering producing such bad results?
body: <p>I have a dataset with 60 000 rows and 19 columns ( I will leave a sample below) and I am trying to make clusters. Using the k-means algorithm I am getting a very low score.</p>
  <p><strong>Dataset Sample</strong></p>
  <p><a href="https://i.stack.imgur.com/9aeA5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9aeA5.png" alt="enter image description here" /></a></p>
  <p>Since some of my columns are categorical variables I proceded to transform them in continous using dictionaries variables as follows:</p>
  <pre><code>def education_dict(data):
      education_dict= {
          &quot;Bachelors&quot;: 0,
          &quot;Graduate Degree&quot;: 1,
          &quot;High School&quot;: 2,
          &quot;Partial College&quot;: 3,
          &quot;Partial High School&quot;: 4
      }
      data[&quot;IDEducation&quot;]=data[&quot;Education&quot;].map(education_dict)
  </code></pre>
  <p>After converting the categorical variable to a continuous variable I delete the old variable.</p>
  <p>After that I do the normalization of the data ( all columns since now all of them are continuous) and I proceed to the k-means algorithm.</p>
  <pre><code>mms=MinMaxScaler()
  mms.fit(data)
  Xnorm=mms.transform(data)
  print(Xnorm.min(axis=0))
  print(Xnorm.max(axis=0))
  print(Xnorm.shape)
  
  km=KMeans(n_clusters=10,n_init=1000,max_iter=800,random_state=42)
  y_kmeans=km.fit_predict(Xnorm)
  
  #Clustering evaluation
  #Silhouette score
  
  #the closest to 1 the better
  silSc=silhouette_score(data,y_kmeans,metric=&quot;euclidean&quot;)
  print(&quot;Silhouette score: &quot; , round(silSc,3))
  print(&quot;\nThese measures need grand truth\n&quot;)
  </code></pre>
  <p>The cluster evaluation is returning me a silhouette score of about 0.08 and this is to low. If I run hieraquical clustering, which is not suited to big datasets like mine, I get a score of about 0.54</p>
  <pre><code>segmentation=[&quot;single&quot;,&quot;average&quot;,&quot;complete&quot;]
  results=[]
  for met in segmentation:
      distance_matrix=linkage(Xnorm,method=met,metric=&quot;euclidean&quot;)
      #Assign cluster lables
      cluster_labels=fcluster(distance_matrix,3,criterion=&quot;maxclust&quot;)
  
      silSc=silhouette_score(data,cluster_labels,metric=&quot;euclidean&quot;)
      print(&quot;Silhouette score: &quot; , round(silSc,3))
  </code></pre>
  <p>Am I doing something wrong?</p>