title: Computation of Kullback-Leibler (KL) distance between text-documents using numpy
body: <p>My goal is to compute the KL distance between the following text documents:</p>
  
  <pre><code>1)The boy is having a lad relationship
  2)The boy is having a boy relationship
  3)It is a lovely day in NY
  </code></pre>
  
  <p>I first of all vectorised the documents in order to easily apply numpy</p>
  
  <pre><code>1)[1,1,1,1,1,1,1]
  2)[1,2,1,1,1,2,1]
  3)[1,1,1,1,1,1,1]
  </code></pre>
  
  <p>I then applied the following code for computing KL distance between the texts:</p>
  
  <pre><code>import numpy as np
  import math
  from math import log
  
  v=[[1,1,1,1,1,1,1],[1,2,1,1,1,2,1],[1,1,1,1,1,1,1]]
  c=v[0]
  def kl(p, q):
      p = np.asarray(p, dtype=np.float)
      q = np.asarray(q, dtype=np.float)
      return np.sum(np.where(p != 0,(p-q) * np.log10(p / q), 0))
  for x in v:
      KL=kl(x,c)
      print KL
  </code></pre>
  
  <p>Here is the result of the above code: <code>[0.0, 0.602059991328, 0.0]</code>.
  Texts 1 and 3 are completely different, but the distance between them is 0, while texts 1 and 2, which are highly related has a distance of <code>0.602059991328</code>. This isn't accurate.</p>
  
  <p>Does anyone has an idea of what I'm not doing right with regards to KL? Many thanks for your suggestions.  </p>