A journey in e-commerce search
This is actually where something like Elasticsearch would work significantly better than Postgres Full Text Search. PFTS is pretty good when you have a small data set and all of the relevant keywords are in the product data, but as OP discovered, it falls pretty far short of the mark when you have anything even remotely complicated.<p>But putting that aside for a moment, most of the effort here actually has nothing to do with which engine you are using to power your search. For this kind of challenge, it doesn&#x27;t matter if you use Elasticsearch, Algolia, or Postgres Full Text Search! Instead, the problem space has a lot more to do with your data and how we can teach a computer to interpret it the way a human would.<p>In the e-commerce&#x2F;search space, one of the biggest challenges is what I call &quot;query understanding,&quot; which is breaking down the user&#x27;s query and trying to understand what they meant. This isn&#x27;t trying to be &quot;smart&quot; and show you things you aren&#x27;t looking for, but rather understand what the underlying intent is and filter on _that_.<p>In OP&#x27;s scenario, you aren&#x27;t searching for things with the word &quot;blue&quot; and &quot;shirt&quot; in it. You&#x27;re actually searching for one thing: a shirt, with a (strong) preference for blue!<p>Most search engines, by default, will do something to the effect of &quot;SELECT * FROM products WHERE anyfield CONTAINS &#x27;blue&#x27; AND anyfield CONTAINS &#x27;shirt&#x27;&quot; and just return everything. But as OP discovered, that might return a product where &quot;blue pants made by the people who brought you your favorite pink shirt,&quot; which definitely isn&#x27;t what you want.<p>Instead, you look at the query: what are we looking for here? We&#x27;re looking for a shirt. So the entity is &quot;shirt&quot; and the modifier is &quot;blue.&quot; Typically &quot;entities&quot; represent &quot;categories&quot; or &quot;product types&quot;, but often are more granular than what we would put in those filters. Thus, when we construct the information architecture for an e-commerce platform, we need to include specific types of data that represent higher-order abstractions of the product that we&#x27;re looking at. Cheezits aren&#x27;t just Cheezits -- they&#x27;re &quot;crackers&quot;, and more specifically, &quot;cheese crackers.&quot; Milk isn&#x27;t just milk, it&#x27;s a beverage and it&#x27;s dairy.<p>We now have this concept of an entity (&quot;shirt&quot;), and a modifier of &quot;blue&quot;, which through various processes we have determined is a color, a theme, or perhaps a pattern. Then we can return a list of all of the shirts, prioritizing the blue ones.<p>Complicating things is this: is navy considered &quot;blue&quot;? What about teal? Turquoise? Cyan? It&#x27;s important to be precise about the colors for categorization, thumbnail, filtering&#x2F;faceting, and UX purposes, but when people type in search queries, they often are not as precise as their brains are.<p>When sorting the results, should we prioritize proper &quot;blue&quot; over navy or teal? If someone types in &quot;shirt&quot; and then clicks the &quot;blue&quot; filter, what are we showing and hiding based on that?<p>We obviously need to be a little bit smarter here....<p>...and now we&#x27;re down this giant rabbit hole of how to optimize search. Clearly, throwing things into a Postgres Full Text Search isn&#x27;t going to cut it. We&#x27;re going to need to leverage more annotated data (garbage in, garbage out -- make sure your product data is properly annotated!), better query understanding, better sorting, and faceting&#x2F;filtering!<p>And, honestly, most of this isn&#x27;t even about whether we&#x27;re using Elasticsearch or Postgres or Algolia -- a lot of this is taking your meatspace brains and thinking about what words really mean, how people type them in, and how to interpret those words into something that returns something meaningful.<p>This ain&#x27;t Google, but it also ain&#x27;t a phone book. There are tons and tons of weird curve balls that will make you think you&#x27;ll never figure it out, even to the very end of your days.<p>Here are a few random examples of how this gets confusing pretty fast:<p>- &quot;chocolate milk&quot; and &quot;milk chocolate&quot; consist of the same words but are two totally different things! (thank you, n-grams)<p>- all &quot;roses&quot; are &quot;flowers&quot;, but not all flowers are roses! Conversely, &quot;garbage bags&quot; and &quot;trash bags&quot; are the same thing! synonyms can make things weird<p>- &quot;cream cheese&quot; is not cream-flavored cheese; it&#x27;s something totally different. If you show me &quot;sharp cheddar cheese&quot; next to &quot;cream cheese,&quot; I might not be happy with those results.<p>- &quot;strawberry yogurt&quot; is entity &quot;yogurt&quot; (not &quot;strawberry&quot; -- even though strawberries are nouns&#x2F;entities, this time it&#x27;s &quot;flavor&quot;), and &quot;strawberry yogurt made with organic milk&quot; is still yogurt, not milk!<p>- people make typos -- how do you detect them? if you get zero results, try a fuzzy search? That might work, but &quot;coke&quot; and &quot;cake&quot; are one transformation away, yet two totally different things<p>- you may not sell &quot;blue shirts with a whale in an ocean,&quot; but you do sell &quot;blue shirts&quot; and &quot;shirts with whales,&quot; which ones do you show and in what order?<p>One could spend years and years trying to perfect query understanding for any particular business. And perhaps even worse, there are no generalizable strategies to solve for everything! My examples above, if it weren&#x27;t obvious, are food-themed because most of my search experience has revolved around data sets involving food. Searching for clothing presents an entirely different set of challenges! For example, a box of Cheerios is a box of Cheerios, but a T-shirt can come in eleven different sizes and sixteen different colors, and each of them are valid search results. Messy!<p>This is a pretty fun topic, IMO. Love to see posts about it on HN and love to see conversations about it!