What every software engineer should know about search
Ex-Google search engineer here, now using hosted ElasticSearch extensively in my startup.<p>This is a really good overview.  If there&#x27;s one part I want to highlight, it&#x27;s that you should <i>expect to spend a lot of time fine-tuning your ranking function for your particular product &amp; corpus</i>.  The default ElasticSearch ranking function kinda sucks.  It was changed in ES 5.0 to Okapi BM25, which is the current academic state-of-the-art in non-machine-learned ranking functions.  However, search is one field where the current academic state-of-the-art is at least a couple decades behind where things are in industry.  When you use a service with good search that just works, chances are that there&#x27;s been a lot of engineer hours devoted to identifying exactly which signals are most useful in your corpus, how they relate to subjective evaluations of relevance, and how to clean them up so that noise doesn&#x27;t dominate the signal.<p>These signals are tightly tied to the corpus you&#x27;re working with.  One slightly unpleasant surprise striking out on my own was just how little the stuff I learned about websearch is relevant to working with different corpora (though I guess I should&#x27;ve predicted this, having seen how many different teams across Google Search operated and how all their signals differed from core websearch).  I&#x27;ll reiterate the article&#x27;s point about <i>process</i> and <i>method</i> being more important than any specific function or algorithm; the most useful stuff I learned at Google was actually a process for taking a vague domain where I&#x27;m not sure what&#x27;s out there, and then learning how to turn that into a system for getting useful information out of it.<p>And then once you&#x27;ve got users, you probably want to feed usage data back into it through some sort of machine learning.  In that order, though; with all the hotness around AI, it&#x27;s really tempting to think &quot;Oh, I&#x27;ll just read the latest LtR papers, Mechanical Turk a bunch of query evaluations, and feed the data into the algorithm&quot;, but without a deeper understanding of the particular corpus you&#x27;re working with and how users want to access it, it&#x27;s unlikely an initial LtR system will perform well enough to attract enough users to bootstrap the system.