Lucene: The Good Parts
Lucene is quite fantastic and Elasticsearch makes it a joy to use.<p>Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some parts with C++ and saw massive gains (but with a disclaimer that this was in no way indicative that Java wasn't fast).<p>I've considered trying to implement something like Lucene in, say, Rust, but then I see just how utterly massive Lucene is. Just the fuzzy search part alone required implementing code to generate code from a Russian PhD thesis they didn't fully understand.[1] So, no matter how many cycles the JVM is needlessly burning, Lucene just seems to advanced to write it without the overhead. (And maybe my intuition is just wrong and the overhead is only a few percent.)<p>1: <a href="http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html" rel="nofollow">http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is...</a>