title: Interpreting the output of SciPy's hierarchical clustering dendrogram? (maybe found a bug...)
body: <p>I am trying to figure out how the output of <code>scipy.cluster.hierarchy.dendrogram</code> works... I thought I knew how it worked and I was able to use the output to reconstruct the dendrogram but it seems as if I am not understanding it anymore or there is a bug in <code>Python 3</code>'s version of this module. </p>
  
  <p>This answer, <a href="https://stackoverflow.com/questions/16883412/how-do-i-get-the-subtrees-of-dendrogram-made-by-scipy-cluster-hierarchy">how do I get the subtrees of dendrogram made by scipy.cluster.hierarchy</a>, implies that the <code>dendrogram</code> output dictionary gives <code>dict_keys(['icoord', 'ivl', 'color_list', 'leaves', 'dcoord'])</code> w/ all of the same size so you can <code>zip</code> them and <code>plt.plot</code> them to reconstruct the dendrogram. </p>
  
  <p>Seems simple enough and I did get it work back when I used <code>Python 2.7.11</code> but once I upgraded to <code>Python 3.5.1</code> my old scripts weren't giving me the same results. </p>
  
  <p>I started reworking my clusters for a very simple repeatable example and think I may have found a bug in Python 3.5.1's version of <code>SciPy version 0.17.1-np110py35_1</code>.  Going to use the <code>Scikit-learn</code> datasets b/c most people have that module from the conda distribution. </p>
  
  <p><strong>Why aren't these lining up and how come I am unable to reconstruct the dendrogram in this way?</strong></p>
  
  <pre><code># Init
  import pandas as pd
  import numpy as np
  import matplotlib.pyplot as plt
  import seaborn as sns; sns.set()
  
  # Load data
  from sklearn.datasets import load_diabetes
  
  # Clustering
  from scipy.cluster.hierarchy import dendrogram, fcluster, leaves_list
  from scipy.spatial import distance
  from fastcluster import linkage # You can use SciPy one too
  
  %matplotlib inline
  
  # Dataset
  A_data = load_diabetes().data
  DF_diabetes = pd.DataFrame(A_data, columns = ["attr_%d" % j for j in range(A_data.shape[1])])
  
  # Absolute value of correlation matrix, then subtract from 1 for disimilarity
  DF_dism = 1 - np.abs(DF_diabetes.corr())
  
  # Compute average linkage
  A_dist = distance.squareform(DF_dism.as_matrix())
  Z = linkage(A_dist,method="average")
  
  # I modded the SO code from the above answer for the plot function
  def plot_tree( D_dendro, ax ):
      # Set up plotting data
      leaves = D_dendro["ivl"]
      icoord = np.array( D_dendro['icoord'] )
      dcoord = np.array( D_dendro['dcoord'] )
      color_list = D_dendro["color_list"]
  
      # Plot colors
      for leaf, xs, ys, color in zip(leaves, icoord, dcoord, color_list):
          print(leaf, xs, ys, color, sep="\t")
          plt.plot(xs, ys,  color)
  
      # Set min/max of plots
      xmin, xmax = icoord.min(), icoord.max()
      ymin, ymax = dcoord.min(), dcoord.max()
  
      plt.xlim( xmin-10, xmax + 0.1*abs(xmax) )
      plt.ylim( ymin, ymax + 0.1*abs(ymax) )
  
      # Set up ticks
      ax.set_xticks( np.arange(5, len(leaves) * 10 + 5, 10))
      ax.set_xticklabels(leaves, fontsize=10, rotation=45)
  
      plt.show()
  
  fig, ax = plt.subplots()
  D1 = dendrogram(Z=Z, labels=DF_dism.index, color_threshold=None, no_plot=True)
  plot_tree(D_dendro=D1, ax=ax)
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/srbnF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/srbnF.png" alt="enter image description here"></a></p>
  
  <pre><code>attr_1  [ 15.  15.  25.  25.]   [ 0.          0.10333704  0.10333704  0.        ]   g
  attr_4  [ 55.  55.  65.  65.]   [ 0.          0.26150727  0.26150727  0.        ]   r
  attr_5  [ 45.  45.  60.  60.]   [ 0.          0.4917828   0.4917828   0.26150727]   r
  attr_2  [ 35.   35.   52.5  52.5]   [ 0.          0.59107459  0.59107459  0.4917828 ]   b
  attr_8  [ 20.    20.    43.75  43.75]   [ 0.10333704  0.65064998  0.65064998  0.59107459]   b
  attr_6  [ 85.  85.  95.  95.]   [ 0.          0.60957062  0.60957062  0.        ]   b
  attr_7  [ 75.  75.  90.  90.]   [ 0.          0.68142114  0.68142114  0.60957062]   b
  attr_0  [ 31.875  31.875  82.5    82.5  ]   [ 0.65064998  0.72066112  0.72066112  0.68142114]   b
  attr_3  [  5.       5.      57.1875  57.1875]   [ 0.          0.80554653  0.80554653  0.72066112]   b
  </code></pre>
  
  <p>Here's one w/o the labels and just the <code>icoord</code> values for the x-axis
  <a href="https://i.stack.imgur.com/X9Vby.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/X9Vby.png" alt="enter image description here"></a></p>
  
  <p>So check out the colors aren't mapping correctly. It says <code>[ 15.  15.  25.  25.]</code> for the <code>icoord</code> goes with <code>attr_1</code> but based on the values it looks like it goes with <code>attr_4</code>. Also, it doesn't go to all the way to the last leaf (<code>attr_9</code>) and that's b/c the length of <code>icoord</code> and <code>dcoord</code> is 1 less than the amount of <code>ivl</code> labels. </p>
  
  <pre><code>print([len(x) for x in [leaves, icoord, dcoord, color_list]]) 
  #[10, 9, 9, 9]
  </code></pre>