Let's build a full-text search engine
>To query the index, we are going to apply the same tokenizer and filters we used for indexing<p>From developing a Lucene-based search engine years ago, I learned a very useful trick for things like stemming and synonyms: <i>don't</i> use the same tokenizer/filters that you used for indexing. It's much more flexible if you instead expand the query terms, replacing a given term with a list of OR clauses with all the synonyms.<p>So during indexing, if the document has "fishing", then you just add that to the index.<p>Later, during querying, the user can search for "fish" with optional stemming enabled, and their query will get re-written into<p>"fish" OR "fishing" OR "fishy"<p>This way, the user has control at query-time, and you can manage your synonym/stemming database without having to rebuild your index.<p>Of course, I had to write a custom parser for this but it was well worth it imo.