<p dir="auto">It would be nice to be able to use the minhash token filter, using <a href="https://issues.apache.org/jira/browse/LUCENE-6968" rel="nofollow">minHash token filter</a> added to Lucene.</p>
<p dir="auto">The minhash algorithm implements locality-sensitive hashing. Basically, every token is hashed into a hash table. The hash table is then sampled to generate a smaller set of representative tokens.</p>
<p dir="auto">This has several applications</p>
<ul dir="auto">
<li>Dimensionality reduction that preserves information that can be used in similarity calculations. For example, efficient deduplication over large sets of text, more like this, etc.</li>
</ul>
<p dir="auto">See elasticsearch-cloud-aws PR 118 from <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/NicolasTr/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/NicolasTr">@NicolasTr</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="42862610" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch-cloud-aws/issues/118" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch-cloud-aws/pull/118/hovercard" href="https://github.com/elastic/elasticsearch-cloud-aws/pull/118">elastic/elasticsearch-cloud-aws#118</a></p>