Show HN: Fast vector similarity using Rust and Python
Author here. I just wanted to add that I struggled mightily to include normalized mutual information in this library but simply couldn’t get it to run quickly enough with 15,000 dimensional vectors to include it (it caused the “all” option to take 20x longer to execute).<p>I was able to get non-normalized MI to run fast, but then you end up with a measure that doesn’t have any intuitive meaning (i.e., it’s not a value in the range [0.0, 1.0] or anything like that— it’s something like 5.2, where this depends on the length of the vectors and the magnitude of the elements).<p>I tried many different approaches, but computing the joint entropy was just too computationally expensive, even using BLAS&#x2F;LAPACK and lots of vectorization. I tried using various kernel density estimation techniques but couldn’t get anything to work well and quickly. I suppose I could just approximate it by taking a random subset of indices like I did for the distance correlation measure, but I find that unsatisfying.<p>So if there are any math&#x2F;Rust wizards in here who have any suggestions, please let me know!