Let's build a full-text search engine
>To query the index, we are going to apply the same tokenizer and filters we used for indexingFrom developing a Lucene-based search engine years ago, I learned a very useful trick for things like stemming and synonyms: don't use the same tokenizer/filters that you used for indexing. It's much more flexible if you instead expand the query terms, replacing a given term with a list of OR clauses with all the synonyms.So during indexing, if the document has "fishing", then you just add that to the index.Later, during querying, the user can search for "fish" with optional stemming enabled, and their query will get re-written into"fish" OR "fishing" OR "fishy"This way, the user has control at query-time, and you can manage your synonym/stemming database without having to rebuild your index.Of course, I had to write a custom parser for this but it was well worth it imo.