title: Using scipy pdist on pandas data frames (and) lists
body: <p>I've run into an odd problem yet again. </p>
  
  <p>Suppose I have the following dummy data frame (by way of demonstrating my problem):</p>
  
  <pre><code>import numpy as np
  import pandas as pd
  import string
  
  # Test data frame
  N = 3
  col_ids = string.letters[:N]
  df = pd.DataFrame(
      np.random.randn(5, 3*N), 
      columns=['{}_{}'.format(letter, coord) for letter in col_ids for coord in list('xyz')])
  
  df
  </code></pre>
  
  <p>This produces:</p>
  
  <pre><code>     A_x         A_y         A_z         B_x         B_y         B_z         C_x         C_y         C_z
  0   -1.339040    0.185817    0.083120    0.498545   -0.569518    0.580264    0.453234    1.336992   -0.346724
  1   -0.938575    0.367866    1.084475    1.497117    0.349927   -0.726140   -0.870142   -0.371153   -0.881763
  2   -0.346819   -1.689058   -0.475032   -0.625383   -0.890025    0.929955    0.683413    0.819212    0.102625
  3    0.359540   -0.125700   -0.900680   -0.403000    2.655242   -0.607996    1.117012   -0.905600    0.671239
  4    1.624630   -1.036742    0.538341   -0.682000    0.542178   -0.001380   -1.126426    0.756532   -0.701805
  </code></pre>
  
  <p>Now I would like to use <code>scipy.spatial.distance.pdist</code> on this pandas data frame. This turns out to be a rather non-trivial process. What <code>pdist</code> does is to compute the distance between m points using Euclidean distance (2-norm) as the distance metric between the points. The points are arranged as m n-dimensional row vectors in the matrix X (<a href="http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.pdist.html" rel="nofollow">source</a>).</p>
  
  <p>So, there are a couple of things that one has to do to create a function that operates on a pandas data frame, such that the pdist function can be used. You will note that pdist is convenient when the number of points gets very large. I've tried making my own, which works for a one-row data-frame, but I cannot get it to work, ideally, on the whole data frame at once. </p>
  
  <p>Here's my attempt:</p>
  
  <pre><code>from scipy.spatial.distance import pdist, squareform
  import numpy as np
  import pandas as pd
  import string
  
  def Euclidean_distance(df):
      EcDist = pd.DataFrame(index=df.index) # results container
      arr = df.values # Store data frame values into a numpy array
      tag_list =  [num for elem in arr for num in elem] # flatten numpy array into single list
      tag_list_3D = zip(*[iter(tag_list)]*3) # separate list into length = 3 sub-lists, that pdist() can work with
      EcDist = pdist(tag_list_3D) # the distance between m points using Euclidean distance (2-norm)
      return EcDist
  </code></pre>
  
  <p>First I begin my creating a results container in pandas form, to store the result in. Secondly I save the pandas data frame as a numpy array, in order to get it into list form in the next step. It has to be list form because the <code>pdist</code> function does only operate on lists. When saving the data frame into an array, it stores it as a list within a list. This has to be flattened which is saved in the 'tag_list' variable. Thirdly, the tag_list is furthered reduced into sub-lists of length three, such that the x, y and z coordinates can be obtained for each point, which can the be used to find the Euclidean distance between all of these points (in this example there are three points: A,B and C each being three dimensional). </p>
  
  <p>As said, the function works if the data frame is a single row, but when using the function in the given example it calculates the Euclidean distance for 5x3 points, which yields a total of 105 distances. What I want it to do is to calculate the distances per row (so pdist should only work on a 1x3 vector at a time). Such that my final results, for this example, would look something like this:</p>
  
  <pre><code>   dist_1    dist_2    dist_3
  0  0.807271  0.142495  1.759969
  1  0.180112  0.641855  0.257957
  2  0.196950  1.334812  0.638719
  3  0.145780  0.384268  0.577387
  4  0.044030  0.735428  0.549897
  </code></pre>
  
  <p>(these are just dummy numbers to show the desired shape)</p>
  
  <p>Hence how do I get my function to apply to the data frame in a row-wise fashion? 
  Or better yet, how can I get it to perform the function on the entire data frame at once, and then store the result in a new data frame?</p>
  
  <p>Any help would be very appreciated. Thanks.</p>