title: Clustering millions of large binary vectors?
body: <p>I want to generate millions of large binary vectors (10_000 ... 100_000 bits).
  Then I want to cluster them by OVERLAP (AND) .
  After that I want to reorder the vectors according to the clustering and save it for later.</p>
  <p>Scipy have a clustering method but probably it wont work on such a large dataset.
  Numpy wont work, cause i will get out of memory.
  Vaex may work but i have to write the clustering algorithm myself and it will be in python, so it will be slow and read/only. Also I'm not sure what format to use when generating the data : csv,hdf5 ??</p>
  <p>Any possible solution ? any other tool or technique ?</p>
  <p>It is also an option to cluster them as i generate them, if that is possible ? I have to do some clever indexing technique !!!</p>